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ABSTRACT 

Smart  Systems  for  Logistics  Command  and  Control  (SSLC2)  is  an  Air  Force 
Research  Laboratory  Warfighter  Readiness  Research  Division  program  to  develop  and 
apply  technologies  to  collect  the  critical  information  required  to  effectively  manage 
logistics  resources  in  support  of  combat  operations.  The  purpose  of  SSLC2  is  to  develop 
capability  requirements  that  employ  technologies  and  techniques  to  autonomously  collect 
and  fuse  critical  data  in  order  to  create  decision  quality  information  and  effectively 
present  information  to  support  cognitive  tasks  performed  by  logistics  and  operations 
decision-makers. 

SSLC2  Spiral  One  was  focused  on  wing-level  personnel  concentrating  on  one 
primary  decision,  the  fix/swap  decision  of  an  aircraft.  The  technologies  developed  under 
this  spiral  can  be  applied  to  many  logistics  and  operations  settings  to  include  the  Space 
ground  asset  management  area.  This  report  documents  the  efforts  associated  with  the 
data  collection,  cognitive  tasks  analysis,  storyboard  development,  and  simulation  test 
system  development,  along  with  the  Scientific  Study  approach,  methods,  results,  and 
recommendations. 

The  Scientific  Study  was  designed  and  conducted  to  validate  the  cognitive  models 
developed  from  the  data  collection  and  cognitive  task  analysis  and  evaluate  user 
performance  and  opinions  on  enhanced  data  streams  demonstrated  in  the  simulation  test 
system.  The  study  was  conducted  across  four  sites  with  each  participant  receiving  all 
three  conditions;  SSLC2  simulated  decision  support  capability,  off-the-shelf  technology 
condition  called  WhereNet  that  provides  Radio  Frequency  Identification  /  Real  Time 
Location  System  (RFID/RTLS)  technology,  and  Baseline  (status  quo). 

The  study  validated  the  hypotheses  that  participants  would  prefer  SSLC2  over  the 
other  technology  conditions  along  with  validating  the  research  objectives.  SSLC2  was 
preferred  over  WhereNet  and  Baseline  for  monitoring  and  locating  resources.  SSLC2 
was  preferred  over  Baseline  for  identifying  resource  availability;  SSLC2  was  preferred 
over  WhereNet  for  tracking  time  related  to  the  movement  of  resources  and  for  making  the 
Fix/Swap  decision.  When  participants  rated  their  agreement  with  a  variety  of  task 
statements  there  were  significant  differences  between  SSLC2  and  Baseline  but  no 
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differences  between  SSLC2  and  WhereNet.  However,  users  commented  that  interaction 
with  WhereNet  was  somewhat  difficult.  The  results  concluded  the  Cognitive  Model  was 
accurate  and  the  field  test  verbal  protocol  results  provided  more  detailed  decision 
strategies. 

Overall,  the  results  show  that  SSLC2  must  consider  how  enhanced  data  streams  (such 
as  location  of  resources)  can  be  integrated  with  existing  information  to  support  user  tasks 
and  decisions  within  the  context  of  their  work.  Location  information  as  a  stand  alone 
system  (WhereNet)  was  not  preferred. 

Detailed  decision  strategies  and  detailed  decision  impact  information  collected  during 
the  Scientific  Study  will  be  incorporated  into  the  next  Spiral  to  expand  and  cover  more 
complex  decision  making.  The  next  Spiral  will  also  expand  on  the  RFID/RTLS 
capabilities  with  enhanced  sensor  technology. 
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1  Summary 

The  Air  Force  Research  Laboratory  Logistics  Readiness  Branch  (AFRL/HEAL) 
contracted  with  the  GRACAR  Corporation  to  develop  and  apply  technologies  to  collect 
the  critical  information  required  to  effectively  manage  logistics  resources  in  support  of 
combat  operations.  Smart  Systems  for  Logistics  Command  and  Control  (SSLC2) 
developed  capability  requirements  that  employ  technologies  and  techniques  to 
autonomously  collect  and  fuse  critical  data  in  order  to  create  decision  quality  information 
and  effectively  present  information  to  support  cognitive  tasks  performed  by  logistics  and 
operations  decision-makers. 

Today’s  logistics  and  operational  environment  has  little  cross  echelon  situational 
awareness.  Data  capture  and  decision  analysis  are  largely  manual  processes.  Many  of  the 
current  systems  are  standalone  and  do  not  share  data.  Personnel  are  often  challenged 
when  trying  to  locate  the  resources  they  need  to  perform  their  job.  With  many  of  the  new 
technologies  they  can  often  be  overwhelmed  with  too  much  data  to  effectively  plan  and 
allocate  logistics  resources.  Personnel  need  a  means  to  identify  the  impact  of  their 
logistics  actions  on  operational  capability.  Often  decisions  are  made  to  optimize  resource 
utilization  and  not  operational  capability,  such  as  sortie  production. 

Technologies  such  as  Radio  Frequency  Identification  /  Real  Time  Location  Systems 
(RFID/RTLS)  and  sensors  improve  situational  awareness  with  respect  to  location  and 
resource  status.  Flowever,  they  do  not  provide  insight  into  the  whole  operational 
capability.  The  decision  making  is  still  limited  by  the  ability  to  assimilate  the  data  into 
reasonable,  actionable  information. 

SSLC2  utilizes  the  advances  in  smart  sensors,  computing  technologies,  information 
networks,  and  researches  the  human  cognition  and  performance  impacts  from  these 
advances.  SSLC2  provides  decision  quality  information  fused  from  all  of  these  enhanced 
data  streams.  Location  and  status  data  provides  near  time  monitoring  capability,  while 
production  and  scheduling  algorithms  equate  resource  availability  to  operation  capability 
and  improved  situational  awareness.  This  provides  decision  makers  insight  into  the 
operational  impacts  of  their  logistics  decisions. 
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SSLC2  started  the  effort  by  first  fully  understanding  the  environment  and  the 
problem.  A  thorough  literature  search  was  performed  and  flightline  personnel  were 
interviewed  across  a  number  of  Air  Force  sites  (active,  guard  and  reserve)  using  a  variety 
of  techniques.  Knowledge  representations  were  developed  from  the  information  to  depict 
the  processes  and  cognitive  models.  Storyboards  were  developed  from  information  from 
these  representations  and  data  collected.  Uses  cases  were  developed  to  start  the 
simulation  test  system  development.  The  simulation  test  system  was  developed,  reusing 
as  much  of  Logistics  Command  &  Information  Support  (LOCIS)  as  possible,  tailoring  to 
the  SSLC2  field  requirements  [7]. 

A  Scientific  Study  was  designed  and  conducted  to  study  the  impacts  of  enhanced  data 
streams  provided  by  RFID/RTLS  technology  and  fused  data  from  legacy  systems  and 
potential  future  automated  capabilities.  The  Scientific  Study  objectives  were  to: 

1)  Validate  the  cognitive  model  developed  from  data  collection  interviews 
through  verbal  protocol  (source)  techniques  and  to  determine  what 
information  is  most  useful  to  help  Expeditors  make  flightline  decisions. 

2)  Evaluate  user  performance  and  opinions  on  enhanced  data  streams. 

The  Scientific  Study  compared  three  conditions.  The  SSLC2  condition  integrated 
RFID/RTLS  technology  with  flightline  information  for  improved  decision  making.  The 
off-the-shelf  technology  condition  called  WhereNet  provided  RFID/RTLS  location  of 
resources  but  did  not  integrate  the  information  with  the  Expeditors  work  and  decision 
processes.  The  Baseline  condition  was  included  as  a  control  condition,  allowing  for 
comparison  of  the  SSLC2  approach  to  current  practice  and  enabling  validation  of  the 
cognitive  model. 

The  Scientific  Study  focused  on  the  fix  or  swap  decision  construct  identified  in  the 
data  collection  and  cognitive  task  analysis  processes.  The  verbal  protocol  technique  was 
used  to  collect  data  related  to  the  types  of  information  and  processes  the  Expeditors  use 
to  make  their  decisions.  Participants  were  also  asked  to  provide  their  opinions  of  the  three 
conditions.  The  hypothesis  was  that  participants  would  prefer  the  SSLC2  approach 
compared  to  the  WhereNet  off-the-shelf  RFID/RTLS  technology  system.  Because 
participants  are  so  familiar  with  current  practice  and  often  resistant  to  change,  it  was 
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hypothesized  that  there  would  be  no  difference  in  preferences  between  the  SSLC2 
approach  and  the  Baseline  current  practice  approach. 

The  experimental  design  was  a  within-subject  full  factorial  design.  Each  participant 
received  all  three  conditions.  Eighteen  people  volunteered  to  participate  in  the  study. 
Participants  were  from  the  445th  Airlift  Wing  Air  Force  Reserve  (AFRES)  at  Wright- 
Patterson  Air  Force  Base  (AFB),  OH;  179th  Airlift  Wing  at  Mansfield  Air  National  Guard 
(ANG),  OFI;  180th  Fighter  Wing  at  Toledo  ANG,  OH;  and  122nd  Fighter  Wing  at  Ft. 
Wayne  ANG,  IN.  Nine  participants  dealt  with  airlift  aircraft,  and  nine  participants 
handled  primarily  fighter  aircraft. 

The  study  was  conducted  by  creating  scenarios  in  which  an  aircraft  on  the  flight 
schedule  had  a  specific  maintenance  problem.  The  participant  acting  as  an  Expeditor 
analyzed  the  problem  and  determined  whether  to  fix  or  swap  the  aircraft.  Time  would 
advance  and  the  participant  would  monitor  the  flightline  and  have  to  assess  and  resolve 
future  problems.  Participants  were  given  questionnaires  to  complete  throughout  the 
evaluation  process.  They  were  given  a  pre-test  questionnaire,  a  post-condition 
questionnaire  after  completing  each  of  the  three  experimental  conditions,  and  a  post-test 
questionnaire  at  the  end  of  the  study. 

The  post  condition  questionnaires  addressed  participants’  assessment  of  each 
technology  by  asking  them  to  rate  their  agreement  on  whether  the  technology  provided 
support  for  monitoring  of  resources,  locating  resources,  identifying  resource  availability, 
tracking  time  related  to  resources,  usability,  usefulness,  impact  on  decision  making 
(including  the  fix/swap  decision),  and  situation  awareness.  When  collapsing  across  all 
statements,  participants  rated  SSLC2  to  be  better  than  Baseline.  Seven  specific 
statements  showed  significant  differences  among  the  three  conditions,  with  all  seven 
statements  showing  a  difference  between  SSLC2  and  Baseline.  Four  of  the  seven 
statements  showed  WhereNet  to  be  better  than  Baseline.  There  was  no  difference 
between  SSLC2  and  WhereNet  for  these  individual  statements. 

At  the  conclusion  of  the  study,  participants  ranked  their  preference  for  the 
technologies.  SSLC2  was  ranked  first  and  was  significantly  different  than  both  WhereNet 
and  Baseline  rankings.  WhereNet  resulted  in  the  lowest  ranking;  however,  it  was  not 
statistically  different  from  Baseline.  Overall,  when  comparing  SSCL2  to  Baseline,  76% 
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of  participants  preferred  SSLC2  for  locating  resources,  65%  preferred  SSLC2  for 
monitoring  the  status  of  resources,  70%  preferred  SSLC2  for  tracking  time  associated 
with  resources  and  58%  preferred  it  for  making  the  fix/swap  decision. 

Post  study  questionnaires  were  also  assessed  for  importance  of  certain  information  on 
decisions,  how  certain  decisions  impact  a  number  of  key  maintenance  metrics,  and  the 
importance  of  data  elements  that  would  be  important  in  an  electronic  decision  support 
tool.  This  information  will  be  utilized  for  expansion  to  complex  multiple  decisions  in 
Spiral  Three. 

A  difference  was  discovered  during  field  test  in  decision  strategies  between  flightline 
airlift  and  fighter  decision  makers.  Therefore,  a  portion  of  the  verbal  protocol  not  only 
assessed  the  data  from  an  overall  perspective,  but  also  analyzed  the  differences  between 
airlift  and  fighter  results. 

During  the  verbal  protocol,  participants  were  asked  to  think  out  loud  during  each 
scenario.  The  team  documented  all  this  information  into  process  steps,  with  each  step 
dissected  into  six  categories,  Information,  Source,  Destination,  Decision,  Time,  and 
Process.  The  data  from  each  of  the  categories  were  analyzed  by  frequency  and  by 
technology.  These  data  were  then  analyzed  as  a  whole  to  validate  the  Cognitive  Model. 
The  order  of  the  process  steps  was  then  dissected  by  airlift  and  fighter  and  the  differences 
analyzed.  These  results  concluded  the  Cognitive  Model  was  accurate  and  the  field  test 
verbal  protocol  results  provided  more  detailed  decision  strategies.  A  cognitive  task 
analysis  was  performed  and  more  detailed  knowledge  representations  were  developed  of 
their  processes  and  cognitive  models. 

Throughout  the  study  observations,  suggestions,  and  feedback  were  documented. 
These  included  differences  in  decision  strategies  between  airlift  and  fighter  showing  the 
benefits  of  conducting  field  tests  at  both  types  of  sites.  While  the  participants  had 
numerous  suggestions  to  changes  to  the  graphical  user  interface  they  liked  having  both  a 
geographic  and  schedule  view  for  whichever  situational  condition  level,  flightline  or 
aircraft  problem,  they  were  working.  Participants  supported  the  RFID/RTLS  technology 
concepts  and  provided  suggestions  for  the  next  steps  with  sensors  and  the  information 
needed. 


4 


The  next  step  for  SSLC2  is  to  expand  the  research  to  investigate  the  impact  of  the 
sensor  technologies  when  users  are  making  multiple  complex  decisions  on  the  flightline. 
All  but  one  participant,  a  retiring  supervisor,  volunteered  to  be  members  of  the  SSLC2 
Users  Group  and  refine  and  expand  the  requirements  for  the  next  field  test  demonstration. 

2  Introduction 

Logistics  support  for  the  U.S.  Military  is  a  complex,  time  dependent,  critical  task. 
Creating  Agile  Combat  Support  (ACS)  for  the  warfighter  requires  real  time  integrated 
information  systems  to  support  human  decision  making.  Through  the  years,  support  tools 
have  been  created  to  support  logistics  problems.  Many  of  the  systems  created  have  been 
standalone  systems  that  do  not  share  data.  In  the  late  1960’s  the  Logistics  Composite 
Model  (LCOM)  was  created  to  allow  Air  Force  leadership  to  evaluate  new  weapon 
systems  or  modifications  impacts  on  logistic  resources  and  model  for  logistics  manpower 
requirements  [3],  Integrated  Maintenance  Information  Systems  (IMIS)  was  developed  in 
the  1990’s  to  integrate  maintenance  information  such  as  technical  manuals,  diagnostic 
instructions,  work  orders,  supply  availability,  etc  in  a  single  hand-held  computer  based 
integrated  system  [14].  In  1999,  AFRL  initiated  the  LOCIS  program  to  research  ways  of 
utilizing  existing  data  from  current  systems  such  as  Core  Automated  Maintenance 
System  (CAMS)  and  IMIS,  and  presenting  the  information  to  the  Operations  Group  (OG) 
Commander,  Logistics  Group  (LG)  Commander,  and  their  respective  staff.  The  focus  of 
the  LOCIS  program  was  on  information  fusion,  decision  support,  and  dynamic  user 
interfaces  [6,  7].  This  successful  program  was  a  first  step  in  developing  integrated 
information  systems  to  support  human  decision  making. 

The  future  vision  is  to  develop  SSLC2  to  fuse  large  amounts  of  location  and  status 
information,  provide  situation  assessment,  and  monitor  equipment  and  human 
performance  in  such  a  way  that  human  operators  and  technology  work  together  as  a 
symbiotic  team.  The  warfighter  requirement  -  the  right  information  delivered  at  the  right 
time  at  the  right  level. 

Advances  in  smart  sensors,  computing  technology,  information  networks,  and 
research  in  human  cognition  and  performance  will  help  us  move  in  this  direction.  Real¬ 
time  sensing  technologies  can  provide  myriads  of  information  to  help  support  Air  Force 
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logistics.  For  example,  RFID  technology  is  being  used  to  improve  In-  Transit  Visibility 
(ITV)  of  equipment  and  supplies  during  deployment  [13].  RFID  tagging  is  becoming 
standard  within  many  commercial  supply  chains.  The  Autonomic  Logistics  System 
(ALS)  is  a  concept  to  allow  automatic  detection  of  aircraft  system  faults  or  system 
deterioration  to  provide  active  logistics  maintenance  information  versus  the  current 
reactive  approach  [5,  11].  War  fighter  physiologic  and  cognitive  monitoring  is  being 
utilized  to  develop  smart  systems  to  adapt  to  the  war  fighters  needs  [8,  12]. 

AFRL  is  interested  in  incorporating  real  time  sensing  technologies  to  improve 
flightline  logistics  support.  This  technology,  as  well  as  others,  has  the  potential  to 
provide  significant  impact  to  flightline  decisions.  Although  off-the-shelf  RFID/RTLS 
technology  provides  immediate  location  information  on  tagged  resources,  and  allows  a 
decision  maker  to  easily  find  these  resources;  in  its  stand  alone  form  it  is  merely  another 
information  stovepipe.  So  while  near  real  time  information  is  available,  it  will  not 
necessarily  improve  flightline  decision  making.  The  challenge  with  sensor  technologies 
is  to  provide  the  correct  information,  at  the  right  time,  in  the  right  format  to  improve 
human  decisions  and  system  performance.  Too  often,  adding  the  technology  without 
considering  the  impact  creates  “shelf-ware.”  Often  development  of  new  technologies, 
such  as  RFID,  occurs  so  rapidly  the  technology  push  can  lead  to  flawed  technology¬ 
centric  systems  lacking  a  human  centered  engineering  approach.  In  the  end  humans  will 
be  the  ones  using  the  technologies  and  systems  to  perform  important  tasks,  make  critical 
decisions  and  fulfill  mission  requirements.  Human  performance  therefore  must  be  central 
to  system  development. 

With  these  considerations  in  mind  the  AFRL  initiated  the  SSLC2  Advanced 
Technology  Demonstration  Program  researching  technologies  and  techniques,  facilitated 
through  software,  to  craft  information  enabling  the  warfighter  to  be  more  agile, 
productive  and  smart  about  critical  logistic  resources  required  for  combat  operations.  It 
has  been  recognized  that  decision  support  tools  must  be  created  to  fully  exploit  real  time 
information.  The  SSLC2  program  focuses  on  human  performance  in  complex,  time 
constrained,  high-stress,  decision  making  situations  and  on  leveraging  technology  to 
assist  the  human  decision-maker  by  capturing  and  displaying  data  in  innovative  ways  to 
promote  more  effective  decision  making  on  the  flightline.  The  overall  goal  is  to  provide 
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decision  support  on  the  flightline  to  improve  sortie  production  resource  allocation, 
personnel  and  resource  scheduling,  and  decision  making  at  multiple  levels. 

2.1  THE  PROBLEM 

Logistics  and  sortie  production  mission  success  depends  on  resources  (people, 
equipment,  and  information).  Yet  few  bases  manage  the  availability  and  performance  of 
their  resources  from  a  mission  perspective.  Resource  management  is  often  driven  by 
priorities  that  are  disconnected  from  the  primary  mission  objectives  and  end-user 
requirements.  Aerospace  Ground  Equipment  (AGE),  test  equipment  and  people  are 
employed  in  stovepipe  fashion,  leading  to  fragmentation  and  “silos”  of  infrastructure. 
This  situation  leaves  managers  with  no  visibility  into  overall  service  levels,  no  way  to 
proactively  detect  and  prevent  availability  and  performance  problems,  and  no  means  of 
quantifying  the  impact  of  downtime  on  the  pilots,  maintainers  and  the  overall  mission.  It 
adds  up  to  delays,  complexity,  over  allocation  and  end-user  frustration. 

The  problem  is  while  there  are  intelligent,  qualified  people,  equipment  information 
including  what  resource  is  needed,  the  resource’s  related  location,  or  its  status  is  not 
always  known.  Equipment  is  used  and  returned  to  the  ready  line  or  the  tool  crib  without 
any  indication  of  a  change  in  status.  Sometimes  equipment  is  used  and  then  left  in  place 
for  future  known  or  unknown  actions.  The  AGE  driver  is  left  to  wonder  if  this  equipment 
is  still  needed,  ready  for  return  to  the  ready  line,  or  in  need  of  servicing  or  repair.  People 
of  the  same  Air  Force  Specialty  Code  (AFSC)  develop  additional  qualifications  or  higher 
levels  of  competency  in  various  tasks,  but  that  information  is  either  documented  in  a 
manual  record,  somewhere  in  a  wide  variety  of  databases  or,  in  the  case  of  levels  of 
competency,  not  documented  at  all.  Many  times  decisions  are  made  to  optimize  resource 
utilization,  not  sortie  production. 

To  meet  the  demand  for  agile  combat  support  a  variety  of  information  technology 
tools  are  being  developed  and  purchased  by  the  U.S.  Air  Force.  Many  of  these  off-the- 
shelf  technologies  are  given  to  users  and  they  are  told  to  figure  out  how  to  use  them.  In 
many  cases  maintenance  personnel  are  developing  their  own  in-house  programs  to  deal 
with  the  information  they  use.  With  the  large  amounts  of  information  and  complex 
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decisions  that  occur  in  the  air  base  environment,  it  becomes  crucial  to  involve  users  in  the 
development  of  technologies,  especially  decision  support  tools,  built  for  their  use. 

RFID  and  RTLS  are  examples  of  off-the-shelf  technologies  with  potential  to  improve 
the  maintenance  process.  RFID  is  sometimes  called  Dedicated  Short  Range 
Communication  (DSRC).  An  RFID  system,  at  a  minimum,  consists  of  three  components: 
an  antenna  and  transceiver  (often  combined  into  one  component  called  a  reader)  and  the 
tag,  of  which  there  are  two  types  (passive  and  active).  In  a  simple  RFID  system  a  passive 
tag  is  used  along  with  a  reader  which  is  usually  a  permanent  gateway  into  a  facility.  This 
will  only  allow  for  entry  and  departure  readings  from  a  defined  facility  or  area.  RTLS 
uses  RFID  technology  to  transmit  the  physical  location  of  RFID  tagged  objects.  RTLS 
systems  require  active  RFID  tags  to  be  attached  to  each  object  needing  tracked  and  Radio 
Frequency  (RF)  transmitters/receivers  located  throughout  the  facility  determine  the 
location  and  send  information  to  a  computerized  tracking  system.  RFID/RTLS 
technologies  with  active  tagging  have  the  potential  to  improve  awareness  related  to 
resource  location  by  showing  where  things  are.  If  the  required  piece  of  equipment  is 
sitting  on  the  flightlinc,  it  is  not  possible  to  tell,  within  the  current  applications,  what  it  is 
doing  out  there.  It  might  be  in  use,  out  of  fuel,  awaiting  a  high  priority  task  to  begin,  or 
ready  for  re-assignment.  These  capabilities  can  be  accomplished  through  customization 
of  the  off  the  shelf  RFID/RTLS  products. 

These  challenges  reflect  up  to  the  Major  Command  (MAJCOM)  and/or  Air  and  Space 
Operations  Center  (AOC)  in  the  broader  context  of  “will  that  base  meet  its  sortie 
schedule  and  support  the  mission?”  and,  “if  not,  what  is  the  impact?”  The  characteristics 
of  the  current  environment  include  little  cross-echelon  situational  awareness  due  to  a 
largely  manual  data  capture  and  decision  analysis  processes.  There  remains  very  little 
insight  to  wing-level  sortie  production  capability.  Decision  making  is  still  limited  by  the 
ability  to  assimilate  data  into  reasoned,  actionable  information.  In  the  end,  off-the-shelf 
Automatic  Information  Technologies  (AIT)  only  solves  a  small  part  of  the  situational 
awareness  and  decision  making  challenge. 
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2.2  SSLC2  PROGRAM  VISION 


The  vision  of  SSLC2  is  to  provide  personnel  decision  quality  information  from 
multiple  data  streams  including  real  time  location  and  status  data  for  monitoring 
capabilities,  as  well  as  algorithms  to  equate  resource  availability  to  operational  capability. 
Decision-makers  will  be  provided  insight  to  the  operational  impact  of  their  decisions.  The 
SSLC2  effort  is  researching  the  fusion  of  technology  with  information  to  provide 
decision  support  and  situational  awareness  aimed  at  demonstrating  the  capability  for 
improved  decision  making  that  can  affect  sortie  generation.  The  concepts  are  applicable 
across  the  AF  spectrum  and  can  easily  be  applied  to  functions  such  as  civil  engineering  or 
transportation.  Functional  systems  that  support  maintenance  data  collection  along  with 
developing  systems  that  will  contain  electronic  Technical  Orders  (TOs),  Enhanced 
Maintenance  Operations  Center  (EMOC)  support  systems,  or  even  direct  maintenance 
support  systems  would  contain  SSLC2  capabilities.  In  this  way,  predictive  logic  can  be 
applied  to  determine  the  appropriate  resources  to  use  and  maintain  the  highest  possible 
Mission  Capable  (MC)  rates.  The  SSLC2  vision  incorporates  Interface  Requirements 
Agreements  (IRA)  with  any  system  that  collects  data  that  would  be  useful  in  the 
management  of  people,  equipment,  and  information.  For  example,  if  medical 
appointments  are  contained  in  a  medical  system,  an  IRA  can  be  drafted  allowing  the 
extraction  of  data  without  violating  the  privacy  of  the  medical  system. 

Another  vision  is  that  a  system  would  contain  job  guides,  the  TO  equivalent  of  a 
“How  To”  guide  for  major  maintenance  tasks  on  each  weapon  system.  These  job  guides 
detail  which  major  pieces  of  equipment  are  required  and  what  follow-on  maintenance  is 
necessary.  In  this  way,  maintenance  management  functions  can  determine  logical 
priority  for  task  accomplishment  with  a  maximized  MC  rate  for  the  end  product. 

SSLC2  should  continuously  monitor  all  assets  in  its  domain  and  report  a  variety  of 
status  indicators.  Some  of  those  indicators  may  be  current  operational  condition  (is  it 
running),  fuel  load  remaining,  built  in  test  status  or  perhaps  data  entered  by  the  last  user. 
This  means  maintenance  managers  will  not  only  know  what  is  needed  for  the  job  but  they 
will,  in  one  look,  know  where  it  is  and  if  it  is  serviced,  repaired  and  available;  in  short  “it 
is  the  right  resource  for  the  job.”  If  the  resource  is  fully  operational  and  ready  for  reuse, 
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the  technician  has  only  to  put  it  back  where  it  belongs  or  have  the  AGE  driver  pick  it  up. 
SSLC2  has  already  assessed  and  published  its  status. 

Information  about  personnel  resources,  such  as  standard  AFSC-related  details,  would 
be  fused  from  the  appropriate  system  and  all  AF  Form  623  and  Job  Qualification 
Standard  (JQS)  data  should  be  available.  In  addition,  maintenance  managers  can  make 
personal  notes  about  workers  to  aid  in  selecting  the  correct  person. 

While  the  vision  is  to  provide  recommendations,  the  actual  decisions  and  selections 
would  be  firmly  in  the  hands  of  the  maintenance  manager.  The  goal  of  SSLC2  is  to 
provide  recommendations  that  optimize  sortie  production  even  if  it  appears  that  resources 
are  sub-optimized. 

Under  the  SSLC2  concept,  not  only  will  maintainers  know  where  things  are,  they  are 
provided  recommendations  as  to  which  resource  is  most  appropriate  for  the  task  at  hand. 
No  longer  will  the  operationally  oriented  decisions  be  the  sole  domain  of  a  few  highly 
experienced  airmen.  While  SSLC2  will  not  make  the  decisions  for  the  maintainer,  it  will 
present  all  the  information  needed  to  make  well  informed  decisions  and 
recommendations. 

Enhanced  situational  awareness  provided  by  the  SSLC2  concept  can  extend  across 
command  echelons  and  provide  MAJCOM  and  Air  Force  Forces  (AFFOR)  decision¬ 
makers  visibility  into  wing-level  operational  capabilities.  They  can  look  down  into  the 
operating  locations  and  determine  independently  if  the  resources  are  in  place  to  meet  the 
flying  schedule  and  Air  Tasking  Order  (ATO).  These  decision  makers  can  also  quickly 
assess  the  impact  of  any  degradation  of  sortie  capacity. 

SSLC2  is  the  logical  conclusion  to  meet  future  needs  of  the  Air  Force.  The  SSLC2 
research  program  is  a  key  project  toward  facilitating  informed  decisions  at  multiple 
levels. 

2.3  PROGRAM  SCOPE 

SSCL2  is  a  6.3  Advanced  Technology  Demonstration  (ATD)  program  researching 
ways  to  make  the  human  warfighter  more  agile,  productive  and  smart  about  the  critical 
logistic  resources  required  to  support  combat  operations.  SSLC2  is  an  effort  to  make 
existing  systems  “smarter”  by  utilizing  data  collection  sensor  technologies  and  fusing 
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data  with  existing  information  systems  to  improve  Logistics  Command  and  Control 
decision  making.  The  program  produces  research,  science  and  methods,  and  evaluates 
how  existing  technologies  can  be  value-added  enhancements  to  existing/forthcoming 
systems.  The  SSLC2  goal  is  to  research,  demonstrate,  and  position  for  transition, 
technology  capabilities  that  can  be  applied  to  and  integrated  into  existing  systems  to 
produce  logistics-based  operational  capability  awareness.  The  end  product  will  identify 
requirements  and  demonstrate  technologies  that  enable  near  real  time  monitoring  of 
information  on  critical  aspects  of  the  logistics  infrastructure.  SSLC2  is  not  generating  a 
new  information  system,  but  generating  requirements  and  specifications  that  can  be 
transitioned  to  an  existing  legacy  system.  Techniques  and  tools  that  can  best  enable 
logistics  decision-makers  to  have  a  high-level  view  of  logistics  operations,  identify 
potential  problems,  and  make  proactive  decisions  at  various  nodes  within  a  logistics 
support  environment  must  be  identified  and  validated. 

2.4  SSLC2  RELEVANCE  TO  AFRL/HE  STRATEGIC  GOALS 

SSLC2  focuses  on  several  of  the  AFRL/HE  strategic  goals  and  objectives,  with  the 
vision  “Unleashing  the  power  of  human  performance  through  technology”  [2]. 

The  Science  and  Technology  Strategy  is  to  “Develop,  integrate,  demonstrate,  and 
transition  affordable  science  and  technology  to  meet  warfighter  needs.”  SSLC2  supports 
the  strategic  goal  to  enable  improved  decision  effectiveness  for  all  warfighters  by 
providing  task  critical  information  portrayal,  and  decision  support  technology. 

The  Technology  Value  Strategy  is  to  deliver  the  value  of  human  effectiveness  science 
and  technology.  SSLC2  supports  multiple  goals  under  this  strategy  including  1) 
Demonstrate  value  and  educate  stakeholders  on  the  relevance  of  human  effectiveness 
technologies;  2)  Provide  timely,  objective,  expert  advice  to  decision  makers;  and  3) 
Collaborate  with  academia  and  industry  to  support  Air  Force  needs  and  to  transfer 
technology. 

The  Transformation  Strategy  is  to  enable  the  Air  Force’s  transformation  from 
platform-centric  programs  to  development  of  effects-based  combat  capabilities.  SSLC2 
supports  the  goal  to  increase  the  directorate’s  capacity  to  meet  current  and  future  Air 
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Force  transformational  technology  development  needs  and  to  expeditiously  transfer 
technology. 

2.5  PROGRAM  SPIRALS 

SSCL2  is  broken  into  three  spirals.  This  report  details  the  outcome  of  Spiral  One. 
This  section  briefly  describes  each  of  the  three  spirals. 

2.5.1  Spiral  One 

Spiral  One  provides  Technology  Availability  Date  (TAD)  1,  for  Autonomous  Data 
Collection  of  Logistics  Resource  Information.  This  spiral  focused  on  the  Expeditor’s 
work  processes.  Expeditors  must  make  decisions  to  allocate  resources  to  keep  aircraft 
maintained  and  operational  to  meet  operational  flying  schedules  (Figure  1).  The  resources 
(e.g.  equipment  and  personnel)  may  be  available  within  their  own  maintenance  units,  or 
resources  may  be  borrowed  from  another  aircraft  maintenance  unit.  It  is  a  challenge  to 
track  resources  and  schedule  information,  and  determine  the  impacts  of  their  decisions 
during  oftentimes  hectic  flightline  operations.  In  the  current  process,  information 
gathering  is  manual  and  paper-based;  while  computer  supported  decision  analysis  tools 
are  minimal  to  nonexistent.  Decisions  are  often  made  quickly  to  solve  the  immediate 
problem,  with  little  thought  to  optimizing  resource  utilization  and  sortie  production. 
Personnel  can  be  overwhelmed  by  multiple  tasks,  frequent  interrupts,  and  the  need  to 
mentally  integrate  information  from  a  variety  of  sources  making  it  difficult  to  effectively 
plan  and  allocate  logistics  resources. 
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Figure  1 :  Characteristics  of  the  Current  Environment 


A  cognitive  task  analysis  was  performed  and  a  cognitive  model  developed  of  the 
Expeditor  to  understand  the  task  domain  and  user  needs.  To  successfully  and  smoothly 
integrate  autonomous  data  collection  and  decision  support  tools  for  the  Expeditor  it  was 
necessary  to  understand  the  Expeditor  task  domain.  Data  were  collected  from  a  variety 
of  sources  and  the  process  and  results  are  described  in  Section  3.1. 


The  autonomous  data  collection  system  investigated  in  this  spiral  is  RFID/RTLS 
technology  for  the  purpose  of  providing  enhanced  data  streams.  For  this  spiral,  an 
RFID/RTLS  system  was  simulated  to  tag  flightline  resources  and  provide  the  Expeditor 
visibility  into  those  resources  when  making  flightline  decisions 

There  were  two  primary  research  objectives  in  this  Spiral: 

Objective  1:  Validate  the  cognitive  model  developed  from  data  collection  during  a 
Scientific  Study.  The  purpose  of  this  objective  was  to  ensure  the  cognitive  model 
preliminary  decision  construct  (model)  was  accurate.  The  Scientific  Study  and  results 
that  describe  the  outcome  of  this  objective  is  presented  in  Section  5.7.1. 
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Objective  2:  Evaluate  user  performance  and  opinions  on  enhanced  data  streams.  A 
Scientific  Study  was  conducted  to  determine  1)  if  users  are  better  able  to  locate  and 
estimate  the  time  it  takes  to  get  all  the  resources  to  the  fix  site,  and  2)  if  users  prefer  the 
SSCL2  integrated  information  approach  to  off-the-shelf  RFID/RTLS  and  current  practice. 
The  Scientific  Study  and  results  that  describe  the  outcome  of  this  objective  is  presented  in 
Section  5.7.2. 

2.5.2  Spiral  Two 

The  Spiral  Two,  TAD  2,  Decision  Improvement  for  SSCL2  will  focus  primarily  on 
the  Space  customer  and  therefore,  will  not  be  addressed  in  this  report. 

2.5.3  Spiral  Three 

Spiral  Three,  TAD  3,  is  Integration  and  Collaboration  and  expands  on  the  Spiral  One 
concepts  by  not  only  supporting  the  Expeditor  making  the  fix/swap  decision,  but  also 
managing  their  flightline  and  flying  schedule  and  having  insight  into  the  impacts  of  their 
decisions  and  resource  utilization  on  the  flying  schedule.  Long  range  sensor  tags  will  be 
added  and  a  site  will  be  outfitted  with  a  RTLS  infrastructure.  The  research  objectives  for 
this  Spiral  focus  on  whether  enhanced  data  streams  impact  complex  multiple  decisions. 

Spiral  Three,  will  demonstrate  and  measure  key  performance  parameters  supporting 
this  research  objective.  Can  they  identify  and  make  their  critical  decisions  in  less  time? 
Do  the  SSLC2  visualization  techniques  improve  their  decision  making  process?  Does 
SSLC2  improve  the  timeliness  of  status  information?  Spiral  Three  will  also  look  at 
impacts  to  global  and  local  situational  awareness. 

2.6  DOCUMENT  OVERVIEW 

The  Spiral  One  Final  Report  presents  the  main  activities  associated  with  the  SSLC2 
program  highlighting  the  research  approach  and  objectives  used  to  gain  insight  into  the 
cognitive  model  of  maintenance  Expeditors  and  to  evaluate  the  effects  of  enhanced  data 
streams  using  RFID/RTLS  technology  on  flightline  decision  making.  The  report  is 
organized  into  sections  describing  the  various  research  activities  of  data  collection, 
design,  development,  and  data  analysis  performed  during  Spiral  One.  Figure  2  illustrates 
the  main  components  of  the  report.  Section  3  presents  the  Spiral  One  Methods  and 
Approach  including  data  collection  and  storyboarding,  software  architecture,  and  the 
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cognitive  task  analysis  activities  used  to  develop  the  Scientific  Study.  Section  4  describes 
the  Scientific  Study  approach.  Section  5  presents  Scientific  Study  analysis  and 
observations.  Section  6  presents  Recommendations  which  includes  future  requirements 
and  considerations  followed  by  conclusions. 

Appendices  are  attached  to  this  report  to  further  illustrate  the  work  accomplished  to 
reach  Spiral  One  goals.  Appendix  A  defines  Acronyms  used  in  this  report.  Data 
Collection  (Appendix  B)  and  Scenarios  (Appendix  C)  highlight  work  done  with  Subject 
Matters  Experts  to  better  understand  the  flightline  environment. 


15 


Section  3  Data  collection,  analysis  and  simulation  development 
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Figure  2:  Document  Overview 

3  Spiral  One  Methods  and  Approach  (Data  collection,  analysis, 
and  simulation  development) 

There  were  two  primary  research  objectives  in  this  Spiral  as  outlined  in  Section  2.5.1. 
To  meet  these  objectives  a  systematic  research  process  was  followed  which  is  illustrated 
in  Figure  3  and  described  below. 
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Figure  3:  SSLC2  Research  Approach 

1.  The  first  step  was  to  perform  a  systems  analysis  focusing  on  the  flightline  wing 
maintenance  process  within  the  overall  sortie  generation  process  with  a  primary 
focus  on  the  role  of  the  Expeditor.  During  this  step  data  were  collected  from 
subject  matter  experts  (SMEs)  at  a  variety  of  Air  Force  sites. 

2.  The  second  step  was  to  compile  the  data  collected  in  Step  1  into  representations  of 
the  information  including  a  cognitive  model  of  Expeditor  tasks. 

3.  The  third  step  was  to  develop  concepts  and  storyboards  that  could  be  used  to 
develop  a  simulation  to  allow  testing  of  SSLC2  concepts  for  enhanced  data 
streams. 

4.  The  fourth  step  was  to  develop  the  software  for  the  simulation  test  system. 

This  section  provides  details  and  findings  for  each  of  the  four  steps  described  above. 
Upon  completion  of  these  steps  a  Scientific  Study  was  designed  and  conducted  to 
validate  the  cognitive  model  (Objective  1)  and  evaluate  the  enhanced  data  streams 
(Objective  2).  The  Scientific  Study  is  described  in  Section  4. 
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3.1  SYSTEMS  ANALYSIS  AND  DATA  COLLECTION 


Data  collection  efforts  focused  on  gathering  information  relevant  to  enhancing 
information  streams  coming  to  flightline  decision  makers.  While  these  data  gathering 
activities  focused  primarily  on  preparation  for  Spiral  One,  data  relevant  to  the  entire 
SSLC2  program  were  collected.  Cross-functional  data  collection  teams  including 
researchers,  analysts  and  subject  matter  experts  collaborated  on  material  preparation, 
participant  visits,  interviews,  and  note  compilation.  When  interviews  were  conducted,  a 
small  cross-functional  team  of  two  to  three  people  were  assigned  to  each  interviewee. 
Depending  on  the  number  of  interviews  scheduled  for  the  site  and  the  time  allotted,  the 
number  of  small  teams  would  fluctuate. 

Participants  in  the  data  collection  activities  were  asked  to  volunteer  approximately  VA 
hours  of  their  time.  The  range  of  personnel  was  very  broad.  Personnel  interviewed 
included  Production  Superintendents  (ProSuper),  Expeditors,  Maintenance  Operation 
Center  (MOC)  controllers,  AGE  supervisors,  AGE  dispatchers,  AGE  drivers,  Fuels 
drivers,  Fuels  Supervisors,  Deployment  Managers,  and  Logistics  Group  Commanders. 

Data  were  collected  from  several  sites.  Each  site  provided  extremely  valuable 
information  for  the  overall  capabilities  of  the  Spiral  One  SSLC2  demonstration.  Initially, 
a  trip  was  made  to  the  Air  Force  Special  Operations  Command  (AFSOC)  at  Ffurlburt 
Field,  FL.  This  trip  provided  initial  information  regarding  resource  allocation  processes 
at  a  high  operational  tempo,  regular  Air  Force,  heavy  aircraft  facility.  A  data  collection 
trip  was  made  to  Luke  AFB,  AZ,  which  is  a  high  operational  tempo  regular  Air  Force 
facility  with  fighter  aircraft.  Luke  AFB  is  one  of  the  Air  Force’s  largest  sortie  production 
bases,  with  its  training  missions. 

In  addition  to  the  regular  Air  Force  sites.  Air  National  Guard  and  AF  Reserve 
Command  facilities  were  also  a  major  part  of  the  data  collection  activities.  Due  to  the 
close  proximity  of  the  445th  Wing  at  Wright-Patterson  AFB,  and  the  Ohio  Air  National 
Guard  base  at  Springfield,  Ohio,  these  sites  were  visited  on  several  occasions  to  collect 
and  verify  information  gathered.  These  sites  provided  not  only  the  full  perspective  of  Air 
Force  capabilities  with  regard  to  flightline  resource  allocation  and  usage;  they  also 
provided  extremely  experienced  personnel  with  deep  knowledge  of  Air  Force  logistics 
challenges.  Along  with  regular  Air  Force  installations,  they  represent  the  full  wartime 
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capability  of  the  Air  Force’s  Joint  Force  Concept  (i.e.,  the  wartime  Air  Force  is 
comprised  of  regular,  guard  and  reserve  personnel). 

Other  data  collection  sites  were  visited  as  opportunities  presented  themselves.  For 
example,  a  trip  was  made  to  Headquarters  Air  Force  Materiel  Command  (HQ  AFMC) 
which  assisted  in  identifying  the  informational  requirements  associated  with  deployment 
activities.  Additionally,  several  meetings  were  held  with  the  Air  Force  Research 
Laboratory’s  Information  Visualization  (AFRL/HECV)  specialists  for  the  purpose  of 
collaborating  on  programs,  and  leveraging  research  methods  and  results  across  programs. 

Methods  used  for  data  collection  included  several  techniques  including  process 
interviews,  personnel  shadowing,  cognitive  walkthroughs,  team  interviews,  and  a  concept 
mapping  interview.  A  primary  focus  of  the  interviews  was  to  understand  difficult  or 
complex  decisions  made  with  regard  to  flightline  resource  allocation  and  usage  and 
decision  process  and  information  used  for  the  fix/swap  decision.  Table  1  identifies  the 
locations,  positions  interviewed,  and  types  of  data  collection  methods  used  at  each  site. 


Table  1:  Data  Collection  Sites  and  Activities 


Hurlburt 

Field 

WPAFB 
HQ  AFMC 

AFRL/ 

HECV 

WPAFB 

Springfield 

OANG 

Luke  AFB 

MOC 

Process 

Process 

Process 

Mx  (Expeditors, 
ProSupers) 

Process 

Process 

Shadowed, 

Process, 

Cognitive 

Interviews 

AGE 

(Dispatchers, 

Supervisors, 

Drivers) 

Process 

Process, 

Cognitive 

Interviews 

Process 

Cognitive 

Interviews 

Shadowed, 

Process, 

Cognitive 

Interviews 

Fuel  (Drivers, 
Supervisors) 

Process 

Shadowed, 

Process, 

Cognitive 

Interviews 

Deployment 

Managers 

Informal 

Team 

Interview 

Process, 

Cognitive 

Interviews 

Process, 

Cognitive 

Interviews 

Researchers 

Informal 

Team 

Interview 

Concept 

Mapping 

19 


3.2  KNOWLEDGE  REPRESENTATIONS 


Four  knowledge  representations  were  generated  from  the  data  collected:  1)  Flow 
chart  and  description  for  flightline  maintenance  process,  2)  Concept  Mapping  of  an 
Expeditor,  3)  Hierarchical  Task  Analysis;  and  4)  Operator  Function  Model  (OFM) 
supplemented  with  COGnitive  information  (OFM-COG).  Each  is  described  below. 

3.2.1  Flightline  Maintenance  Process  Overview 

Figure  4  is  a  basic  flow  diagram  illustrating  the  overall  flight  maintenance  process. 
The  initiation  of  the  maintenance  process  differs  depending  on  whether  the  unit  is 
operating  under  AOC  direction  or  operating  under  training  guidelines.  Sortie  generation 
is  an  iterative  process  and  there  is  no  clear  beginning  or  ending  point.  Maintainers 
generally  consider  the  start  of  the  actual  generation  of  the  day’s  sorties  as  the  beginning 
of  the  process,  but  it  would  be  just  as  appropriate  to  deem  it  to  begin  as  the  aircraft  return 
from  the  day’s  last  flight.  These  landing  aircraft  become  the  basis  for  planning  the  next 
day’s  flying  schedule  based  on  their  maintenance  status. 

When  aircraft  return  from  a  mission  the  aircrew  identifies  the  aircraft  maintenance 
status  code  (AKA  a  “Squawk”).  A  Code  1  squawk  equates  to  a  Fully  Mission  Capable 
(FMC)  aircraft.  Code  2  indicates  a  flyable  aircraft  but  some  system  has  a  problem;  a 
Code  2  aircraft  may  be  FMC.  Code  3  indicates  an  aircraft  with  a  non  flyable  condition. 
Code  2  or  3  squawks  usually  include  verbal  information  indicating  which  aircraft  system 
has  a  problem.  For  example,  a  squawk  of  “Code  3  hydraulics”  would  indicate  a  major 
fault  with  the  hydraulic  system  that  will  ground  the  aircraft  until  repairs  are  made. 
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Figure  4:  Flightline  Maintenance  Process 
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The  Expeditor  assigned  to  the  aircraft  will  record  the  squawked  status  on  their  fleet 
status  board  and  pass  the  status  on  to  the  MOC  and  the  ProSuper.  This  step  is  sometimes 
reversed  where  the  pilot  squawks  over  the  aircraft  radio  and  provides  the  data  to  his 
operational  unit  who  passes  it  to  the  MOC  who  then  passes  it  on  to  the  ProSuper  and 
Expeditor.  Once  the  aircraft  lands  the  aircrew  reports  to  the  debriefing  facility  (AKA 
debrief)  to  discuss  the  aircraft  maintenance  issues  or  “write-ups”  with  the  appropriate 
maintenance  technician.  This  process  helps  to  further  define  the  aircrew’s  “write-up” 
which  is  only  a  short  paragraph  logged  into  the  Aircraft  Air  Force  Technical  Order 
(AFTO)  Form  781  binder  (AKA:  The  Forms).  Typically  the  problem  is  much  more 
complex  than  the  available  space  in  the  13  lines  of  the  AFTO  Form  781a  Discrepancy 
block.  Occasionally  the  problem  is  assigned  to  a  different  system  and  maintenance 
specialty  from  the  original  squawk  based  on  the  debrief  information.  After  debrief,  the 
Aircraft  Forms  are  passed  to  the  Expeditor  who  transfers  the  entries  into  one  line 
summaries  on  the  fleet  status  board  and  returns  the  forms  to  the  proper  aircraft.  The 
Expeditor  then  decides  the  priority  of  which  problem  needs  to  be  worked  by  which 
specialist  on  which  aircraft.  Often  there  is  no  need  to  consider  priority  as  there  are  fewer 
problems  to  work  than  specialists  in  the  relevant  specialty.  In  the  event  that  multiple 
problems  exist  for  a  given  specialty,  a  decision  will  have  to  be  made  based  on  overall 
priorities  established  by  the  ProSuper  and/or  Aircraft  Maintenance  Unit  (AMU) 
supervision. 

As  soon  as  the  squawks  are  confirmed  in  debrief  the  ProSuper  compares  aircraft 
status  changes  to  the  schedule.  The  ProSuper  works  in  concert  with  the  Expeditor  and 
AMU  supervision  to  nominate  aircraft  against  the  schedule  “lines”  (requirements).  Both 
the  remainder  of  the  current  day’s  flying  schedule  (in  execution)  and  the  requirements  for 
the  following  day  have  to  be  considered.  At  the  same  time,  the  Expeditor  is  trying  to 
determine  if  the  broken  aircraft  will  be  available  for  next  scheduled  flight(s). 

On  the  morning  of  the  flying  schedule  execution  Crew  Chiefs  ready  the  aircraft, 
oversee  servicing,  and  perform  the  required  “preflight”  inspection.  As  issues  and  errors 
are  discovered,  the  Expeditor  ensures  the  appropriate  maintenance  technicians  are 
dispatched  to  make  corrections.  Soon  maintenance  actions  take  on  the  frenzied  “red  ball” 
status.  According  to  Air  Force  Instruction  (AFI)  21-101  [1],  “...“Red  Ball”  maintenance 
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normally  occurs  two  hours  prior  to  launch  and  until  aircrew  have  released  the  aircraft 
back  to  maintenance.”  It  is  not  unusual  to  have  a  maintenance  technician  working  a  red 
ball  over  the  lap  of  a  pilot  just  minutes  before  (or  after)  scheduled  launch  time. 

If  no  problems  are  encountered  the  engines  are  started,  taxi  and  takeoff  follow 
according  to  schedule.  However  if  a  problem  is  encountered  the  ProSuper  and  Expeditor 
must  decide  if  there  is  enough  time  to  resolve  the  problem  before  scheduled  takeoff.  If 
the  resolution  or  “fix”  time  is  expected  to  be  within  the  schedule  window,  then  the 
aircraft  goes  to  Fix.  If  not,  the  ProSuper,  Expeditor  and  operations  personnel  make 
considerations  to  move  the  aircrew  to  the  spare  aircraft  (if  one  exists).  If  no  spare  aircraft 
exists  and  the  mission  is  cancelled,  many  times  a  second  aircraft  will  “sympathy  abort” 
because  doctrine  does  not  permit  one  fighter  to  proceed  alone.  Depending  on  the  number 
of  aircraft  launching,  the  assigned  mission  and  the  remaining  pilot’s  qualifications,  the 
second  aircraft  may  join  with  other  aircraft  to  complete  its  mission. 

3.2.2  Cognitive  Task  Analysis  Representations 

The  goal  of  developing  a  cognitive  model  is  to  understand  the  Expeditor’s 

information  needs,  decisions,  goals  and  strategies  for  conducting  their  job.  The  cognitive 
model  provides  a  foundation  for  SSLC2  development.  The  data  collected  for  the 
development  of  the  cognitive  model  includes  the  original  interviews  conducted  at  the 
beginning  of  Spiral  One,  interviews  with  the  SMEs  from  AFRL,  and  a  concept  mapping 
with  one  SME.  These  data  are  presented  using  a  hierarchical  task  analysis,  a  concept 
map,  and  OFM-COG  models  [9],  The  results  of  each  are  presented  in  the  sections  below 
and  diagrams  are  documented  in  Section  2.3  of  Appendix  B:  Data  Collection.  The 
information  from  this  analysis  is  used  to  identify  opportunities  for  providing  the  right 
type  of  support  in  SSLC2. 

3.2.2.1  Concept  Map 

Researchers  met  with  a  SME  to  elicit  knowledge  about  the  Expeditor’s  task  using  a 
concept  map  approach.  A  concept  map  represents  elicited  knowledge  as  a  series  of  nodes 
(concepts)  connected  by  lines  representing  the  relationships  between  the  concepts.  The 
concept  map  is  built  interactively  as  the  expert  describes  their  tasks  and  their 
understanding  of  the  domain.  This  technique  helps  to  organize  the  domain  concepts  and 
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to  transfer  the  information  from  the  expert  to  the  design  team  [4,  10,  and  15].  The 
concept  map  is  presented  in  Figure  5.  Blocks  outlined  in  Red  indicate  tasks  carried  out  by 
the  Specialist,  not  the  Expeditor. 
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Figure  5:  Expeditor  Concept  Map 


Z.2.2.2  Hierarchical  Task  Analysis 

The  primary  role  of  the  Expeditor  is  to  monitor  flightline  maintenance  and  to  expedite 
or  assist  maintenance  so  flights  can  meet  their  sorties.  They  facilitate  the  assignment  and 
movement  of  resources  (equipment  and  people)  to  meet  the  maintenance  needs.  Table  2 
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provides  a  description  of  major  tasks.  Spiral  One  concentrated  on  Tasks  1-  6,  which  are 
presented  in  the  hierarchical  task  diagram  contained  in  Section  2.3.1  of  Appendix  B. 
These  diagrams  illustrate  the  tasks  at  a  high  level. 


Table  2:  Expeditor/ProSuper  Task  List 


Task  Name 

Description 

1.  Prepare  for  Status  meeting 
(start  of  day) 

Reviews  information  related  to  the  current  status  of  the  flightline, 
aircraft,  personnel,  schedules 

2.  Monitor  flightline 
maintenance 

Monitors  the  flightline,  receives  status  updates,  requests  needed 
resources 

3.  Monitor  aircraft  landing 

Monitors  aircraft  landing,  determines  squawk  code  and  possible 
problems 

4.  Launch  aircraft 

Monitor  and  document  aircraft  launch  from  air  crew  show  to  take  off 

5.  Assess  aircraft  status 

Assess  aircraft  state  and  determine  aircraft  status  (FMC,  PMC,  NMC) 

6.  Evaluate  fleet  health 

Evaluate  overall  health  of  aircraft,  review  maintenance  metrics 

3.2.2.3  Operator  Function  Model  and  Cognitive  Information  (OFM-COG) 

An  OFM-COG  was  created  to  present  the  Expeditor’s  Cognitive  Model  [9].  For  the 

OFM  4  sub-functions  of  the  Monitor  Flightline  Maintenance  were  identified:  Monitor 
aircraft  launch,  Monitor  aircraft  landing,  Assess  aircraft  status,  and  Evaluate  fleet  health. 
Information  related  to  evaluating  fleet  health  was  not  collected.  Section  2.3.2  of 
Appendix  B  contains  the  four  sub-functions  using  the  OFM  technique  and  the  OFM-COG 
analysis  of  each  sub-function.  The  definitions  for  the  Task  agents  specified  in  the  tables 
can  also  be  found  in  Section  2.3.2  in  Appendix  B. 

3.2.2.4  Summary  of  Expeditor  Tasks 

The  primary  role  of  the  Expeditor  is  to  monitor  flightline  maintenance  and  to  expedite 
or  facilitate  maintenance  so  flights  can  meet  their  sorties.  Personnel  who  are  selected  to 
be  Expeditors  have  typically  worked  on  the  flightline  as  Crew  Chiefs  and/or  Specialists. 
They  have  several  years  if  not  many  years  of  experience  with  the  flightline  and  specific 
maintenance  issues.  An  Expeditor  is  required  to  be  out  on  the  flightline  at  all  times.  Their 
office  is  in  their  vehicle,  which  they  use  to  move  equipment  and  people,  with  little  time 
available  for  data  input.  They  use  clip  boards  with  paper  to  store  and  retrieve  information 
and  rely  on  radios  for  communication  and  to  help  them  maintain  situation  awareness. 
The  environment  can  be  described  as  high  stress,  especially  when  deployed  where  they 
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often  work  twelve  hour  shifts.  The  Expeditor  is  interrupt  driven,  and  must  juggle  many 
tasks  at  one  time. 

The  Expeditor’s  primary  focus  is  to  ensure  sorties  are  met.  They  review  daily,  weekly 
and  monthly  schedules  and  use  paper  schedules  on  which  they  make  notes.  The  schedules 
and  reports  used  vary  among  maintenance  units  and  bases  and  are  usually  created  in  a 
spreadsheet  file  and  updated  at  the  beginning  and  end  of  a  shift. 

At  the  beginning  of  a  shift  the  Expeditor  meets  with  the  ProSuper  and  personnel  from 
the  previous  shift  (Expeditors,  ProSuper,  Scheduler)  where  he  is  updated  on  all  aircraft 
status  and  maintenance  phases.  They  are  also  updated  on  personnel  information.  Any 
issues  or  problems  are  described  during  the  meeting.  This  meeting  is  essential  to  provide 
the  Expeditor  with  current  situation  assessment.  The  Expeditor  reports  to  the  ProSuper 
and  updates  the  ProSuper  throughout  the  day.  In  some  cases  the  ProSuper  is  also  the 
Expeditor.  Decisions  at  start  of  shift  may  include  what  personnel  to  assign  to  various 
tasks  based  on  who  is  available  and  skill  sets.  They  may  also  make  recommendations  to 
the  ProSuper  to  change  aircraft  tails  to  sorties  based  on  updated  aircraft  maintenance 
stage,  sortie  schedule,  or  supply  availability  information. 

Throughout  the  shift  Expeditors  arc  assessing  aircraft  status  and  maintenance  stages. 
They  monitor  all  aircraft  taking  off  and  landing.  In  addition  to  the  primary  tasks  listed  in 
Table  2,  the  Expeditor  is  also  responsible  for  ensuring  safe  practices  and  AFIs  are  being 
followed. 

The  Expeditor  makes  a  decision  to  fix  or  swap  an  aircraft  in  order  to  meet  a  sortie  and 
provides  their  fix/swap  recommendation  to  the  ProSuper  or  maintenance  supervisor,  who 
has  final  say.  They  may  also  recommend  to  delay  or  cancel  the  sortie.  A  swap  aircraft 
may  be  a  spare,  or  another  aircraft  on  the  flying  schedule.  They  may  also  make  decisions 
to  pull  human  or  equipment  resources  from  one  aircraft  to  another  in  order  to  meet  the 
schedule.  They  focus  on  meeting  the  schedule  and  trying  to  limit  interruptions  to  other 
on-going  aircraft  maintenance  tasks.  Expeditors  often  pull  resources  they  can  see  within 
their  immediate  area.  Decisions  are  made  quickly,  relying  on  previous  experience.  They 
rely  on  knowing  what  stages  aircraft  are  in  and  where  personnel  are  assigned.  They  often 
call  for  resources  throughout  the  day. 
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3.3  STORYBOARD  DEVELOPMENT 

Leveraging  information  captured  during  data  collection  and  knowledge 
representation  stages,  a  series  of  concepts  and  storyboards  were  created  and  used  as  input 
into  the  Graphical  User  Interface  (GUI)  design  as  well  as  the  software  use  cases  for 
development  of  the  simulation  test  system  (see  section  3.4  for  discussion  of  software). 

The  cognitive  task  analysis  indicates  a  basic  workflow  that  can  be  described  as 
follows: 

1 .  Pre-problem  flightline  monitoring; 

2.  Assessment  of  a  specific  problem  occurring  on  the  flightline; 

3.  Weighing  trade-offs  with  regard  to  fixing  or  swapping  the  aircraft;  and 

4.  Executing  the  decision  and  monitoring  progress  toward  that  resolution; 

Using  this  task  structure,  several  scenarios  were  created  to  follow  this  flightline 
decision  making  process. 

To  develop  the  GUI  the  information  needed  by  the  Expeditor  and/or  ProSuper  was 
determined  and  a  framework  for  the  work  process  was  identified.  The  framework 
established  screen  locations  for  the  information  to  be  displayed.  The  framework  was 
modified  from  a  framework  used  during  the  LOCIS  project.  Figure  6  shows  the  basic 
framework.  Using  this  framework,  all  screen  designs  for  the  SSLC2  simulation  test 
system  were  created.  For  any  given  screen,  the  information  elements  were  mapped  to 
their  appropriate  areas  using  the  framework.  Figures  7  and  8  illustrate  two  of  the  GUI 
screen  designs.  The  left  side  of  the  screen  shows  the  workflow  process  steps  (e.g.  assign 
assessor,  perform  assessment,  and  verify  resources).  The  central  portion  of  the  screen 
provides  schedule  and  geographic  views.  These  views  can  be  changed  from  an  overview 
of  the  entire  flightline,  to  a  problem  view  focusing  on  one  aircraft. 
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Figure  6:  SSLC2  Screen  Design 
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Figure  8:  SSLC2  Schedule  View 


3.4  SIMULATION  TEST  SYSTEM  DEVELOPMENT 

The  purpose  of  the  simulation  test  system  was  to  provide  the  ability  to  test  as  well  as 
demonstrate  SSLC2  concepts.  This  section  briefly  details  the  overall  software 
architecture  followed  by  more  detailed  information  including  software  reuse  from  the 
LOCIS  program,  use  case  specifications,  requirements  and  model  traceability,  activity 
and/or  design  model,  algorithms  and  business  logic,  and  database. 


3.4.1  Software  Architecture 

SSLC2  is  a  three-tier  thin  client  web  application.  The  client  tier  is  a  thin  client  web 
application  that  incorporates  the  GUI.  The  only  commercial  software  requirement  for  the 
client  machine  is  Microsoft  Internet  Explorer  version  5  or  greater  and  a  Java  Virtual 
Machine  version  1.4  or  greater.  All  the  GUI  objects  are  downloaded  to  the  client  the  first 
time  the  computer  accesses  SSLC2.  Subsequent  client  requests  result  in  download  of  data 
only.  This  design  reduces  the  bandwidth  requirements  and  provides  a  responsive  system. 
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The  middle  tier,  the  business  logic  of  SSLC2,  is  comprised  of  servlets  currently 
developed  and  hosted  on  WebLogic  version  7.0;  however,  SSLC2  could  be  rehosted  to 
run  on  any  web  server  that  supports  Java  server  pages,  servlets  and  applets. 

Communication  to  the  3rd  tier,  the  database,  is  via  industry  standard  Java  Database 
Connectivity  (JDBC)  protocol.  The  database  used  is  Oracle  9.0.2  and  can  be  hosted  on 
the  same  server  as  the  middle  tier  web  server  or  on  a  separate  server. 

The  only  other  software  interface  created  in  Spiral  One  was  an  interface  to  the 
WhereNet  server  which  provides  the  simulated  resource  location  data.  The  WhereNet 
server,  a  separate  server  from  the  SSLC2  server,  supports  communications  via  HyperText 
Transport  Protocol  (HTTP).  A  server  side  process  was  created  that  subscribes  to  events 
on  the  separate  WhereNet  server  via  HTTP,  in  order  to  receive  location  information. 
This  information  is  retrieved  by  SSLC2  by  pinging  the  WhereNet  Application 
Programming  Interface  (API)  at  a  timed  interval  and  then  inserted  into  the  SSLC2 
database  via  JDBC. 

3.4.1 .1  LOCIS  Reuse 

Capitalizing  on  the  previous  AFRL/HEAL  project,  LOCIS,  SSLC2  was  able  to 
produce  a  thin  client  web  application  prototype  for  the  Spiral  One  Scientific  Study.  This 
section  discusses  the  major  software  components  of  LOCIS  reused  in  SSLC2. 

The  major  LOCIS  reuse  centered  on  the  software  framework.  These  elements 
included:  database  tables,  servlets,  data  objects,  communications  object  used  to  transmit 
data  to  and  from  the  server,  the  main  applet  window,  and  navigation  elements  of  the  main 
element.  The  framework  elements  are  those  objects  and  tables  used  to  deliver  views.  In 
order  to  accommodate  SSLC2  design  elements,  the  GRACAR  team  changed  some 
framework  elements  to  incorporate  Roles.  Roles,  briefly  discussed  in  section  3.4. 1.6, 
required  framework  redesign  to  accommodate  the  Selector  Area  on  the  left  side  of  the 
applet  window. 

The  original  LOCIS  geographic  view  was  reused  to  provide  the  SSLC2  Geographic 
view.  Changes  to  this  element  involved  expanding  the  aircraft  icon  object  to  support 
other  types  of  equipment.  The  capability  for  icons  to  “stack”  was  an  enhancement  to  the 
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original  LOCIS  geographic  view.  Elements  of  the  geographic  view  were  also  reused  in 
the  resource  select  screen. 

Spiral  One  did  not  include  the  exact  LOCIS  flying  and  maintenance  schedules; 
however  elements  of  the  LOCIS  schedule  view  were  used  to  create  the  SSLC2  problem 
schedule  view. 

A  fourth  element  of  LOCIS  reuse  is  the  development  environment.  The  build 
environment  included  batch  files  and  Apache  Ant  build  scripts  that  compiled  the  source 
files  and  created  a  deployment  directory  containing  the  entire  web  application.  This 
allowed  for  a  simple  “drag-and-drop”  development  method  to  deploy  updates  to  the 
SSLC2  application. 

Generally,  the  decision  to  reuse  LOCIS  elements  or  not  was  based  on  SSLC2 
functionality  requirements.  The  development  approach  was  to  strip  LOCIS  down  to  the 
basic  framework  and  then  reintroduce  elements  meeting  SSLC2  needs. 

3.4.1 .2  Use  Case  Specifications 

A  use  case  describes  a  sequence  of  actions  a  system  performs  yielding  an  observable 
result  to  a  particular  actor.  A  series  of  use  cases  were  created  describing  how  SSLC2 
interacts  with  the  Expeditor.  The  use  cases  describe  each  screen  giving  a  basic 
description,  flow  of  events,  alternative  flows,  special  requirements,  preconditions,  post 
conditions  and  any  additional  actors.  The  use  cases  illustrate  exactly  how  SSLC2  will 
respond  to  input. 

3.4.1. 3  Requirements  and  Model  Traceability 

Essential  to  documenting  the  SSLC2  requirements  is  the  capability  to  trace  from  the 

source,  such  as  a  literature  search  documents,  interviews,  previous  research  efforts, 
through  the  Use  Cases  to  the  software  components  and  test  cases.  An  initial  literature 
search  of  logistics  command  and  control  efforts  and  previous  ALRL  research  efforts 
generated  a  list  of  requirements  that  were  then  validated  and  expanded  during  data 
collection  utilizing  the  SME  generated  scenarios.  The  requirements  were  then  grouped 
into  display,  schedule,  parts,  interface,  reporting,  resources,  and  problem  requirements. 
Use  Case  specifications  were  developed  from  these  requirements  and  linked  to  specific 
software  components  and  then  test  cases. 
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3.4.1 .4  Activity  and/or  Design  Model 

For  Spiral  One,  SSLC2  begins  when  a  problem  is  already  identified,  assessed  and 
verified.  Task  steps  for  fixing  or  swapping  the  aircraft  and  times  and  types  of  resources 
are  known  and  in  the  system.  SSLC2  assigns  resources  based  on  proximity  to  the 
problem  aircraft.  Using  SSLC2  the  Expeditor  monitors  the  flightline  and  is  alerted  when 
the  problem  is  entered  into  the  system.  The  Expeditor’s  first  step  is  to  Verify  the 
Resources.  SSLC2  displays  the  technical  order  tasks,  times  for  each  task,  and  resource 
times  to  site.  The  Expeditor  reviews  the  resources  assigned  by  SSLC2  for  both  the  fix  and 
the  swap  solutions.  The  Expeditor  can  override  recommendations,  select  any  additional 
resources  or  manipulate  times  based  on  experience. 

The  scenario  concludes  with  the  Expeditor  selecting  a  course  of  action.  His  options 
are  to  fix  or  swap  the  aircraft  or  delay  or  cancel  the  sortie.  If  fix  is  selected,  the  SSLC2 
concept  is  to  automatically  notify  resource  managers,  technicians,  and  inform  the 
ProSuper.  If  swap  is  selected,  it  is  considered  a  recommendation  that  goes  to  the 
ProSuper  for  approval.  The  final  option  is  to  recommend  delay  or  cancel.  This  also 
becomes  a  recommendation  to  the  ProSuper  who  in  turn  negotiates  a  course  of  action 
with  Operations.  In  SSLC2,  the  problem  information  as  well  as  the  potential  solutions 
would  be  presented  to  the  ProSuper  to  facilitate  the  negotiations. 


3.4.1 .5  Algorithms  and  Business  Logic 

The  SSLC2  simulation  uses  three  algorithms  to  determine  1)  AGE  availability,  2) 
which  aircraft  to  swap,  and  3)  resource  transit  times.  The  logic  for  these  algorithms  is 
described  below. 

1.  AGE  Availability  -  The  specific  location  of  resources  was  specified  in 
WhereNet.  In  this  system  it  is  possible  to  define  zone  labels  within  the  RTLS 
coverage  area.  The  zone  information  is  encoded  within  the  RFID  tag  and  sent  in 
the  message  to  SSLC2.  Zones  were  defined  for  the  entire  flightline,  each  parking 
location,  AGE  yard,  ready  lines,  etc.  In  order  for  SSLC2  to  consider  a  piece  of 
AGE  as  being  available  and  therefore  recommend  it’s  use,  it  had  to  be  FMC  and 
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in  any  available  zone.  AGE  in  any  other  location  was  available  to  be  manually 
selected. 

2.  Candidate  Swap  Aircraft  -  SSLC2  not  only  recommends  resources  to  meet  a  fix 
requirement  but  also  recommends  resources  to  meet  a  tail  swap  requirement. 
One  of  the  resources  needed  is  a  candidate  tail  for  the  swap.  SSLC2  used  the 
following  logic: 

•  If  there  is  a  aircraft  designated  as  a  spare  on  the  flying  schedule  then 
that  is  the  candidate  swap; 

•  If  there  is  not  a  spare  designated,  the  candidate  tail  is  the  last  flyer  on 
that  day;  or 

•  If  no  other  flyers  are  available,  then  select  an  FMC  aircraft. 

3.  Resource  transit  times  -  A  rudimentary  algorithm  was  included  to  compute 
resource  transit  times.  This  algorithm  is  a  straight  line  path-of-travel  along  with 
discrete  radii  regions  (e.g.  a  truck  which  is  between  2  km  and  3  km  away  will 
take  8  minutes  to  travel  that  distance).  Travel  speeds  used  were:  Trucks:  15  mph, 
AGE:  5  mph,  and  Personnel:  2  mph. 

3.4.1 .6  Database 

The  majority  of  the  database  design  was  inherited  as  a  bi-product  of  reuse  of  LOCIS. 
However,  there  were  some  significant  changes  from  LOCIS  in  order  to  accommodate 
SSLC2.  Changes  included  design  elements  such  as  Roles,  storing  AGE/personnel 
information,  and  aircraft  problems  that  result  in  a  fix  or  swap.  The  aircraft  problem 
requirement  generated  the  need  to  be  able  to  associate  tasks,  resources,  and  times  to  both 
the  fix  and  swap  options. 

The  roles  concept  equates  to  a  specific  set  of  software  capabilities  that  are  available  to 
different  types  of  users.  The  capabilities  relate  to  the  types  of  views  the  user  is  able  to 
access. 

The  original  LOCIS  database  was  designed  to  handle  aircraft  data  elements.  SSLC2 
expanded  on  this  concept  by  manipulating  tables  to  add  equipment  and  personnel  data 
elements. 
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The  SSLC2  database  also  contains  information  related  to  aircraft  problems.  Database 
elements  include: 

•  Tasks  necessary  to  fix  the  broken  aircraft; 

•  Tasks  necessary  to  swap  from  the  broken  aircraft  to  a  spare  for  the  sortie; 

•  Time  necessary  to  accomplish  each  task; 

•  Resource  types  necessary  to  accomplish  each  task;  and 

•  Specific  resources  assigned  to  the  tasks. 


4  Scientific  Study  Approach 

4.1  OBJECTIVES 

To  evaluate  enhanced  data  streams  provided  by  RFID/RTLS  technology  a 
Scientific  Study  was  undertaken.  The  study  was  designed  to  compare  three  conditions. 
The  SSLC2  approach  which  integrates  RFID/RTLS  technology  with  flightline 
information  for  improved  decision  making  was  compared  to  an  off-the-shelf  technology 
that  provides  RFID  tags.  The  off-the-shelf  technology  called  WhereNet  provides  location 
of  resources  but  does  not  integrate  the  information  with  the  Expeditors  work  and  decision 
process.  In  addition  to  these  two  conditions,  a  Baseline  condition  was  included  as  a 
control  condition.  This  allows  for  comparison  of  the  SSLC2  approach  to  current  practice 
and  also  enables  validation  of  the  cognitive  model. 

The  Scientific  Study  objectives  were: 

1)  To  validate  the  cognitive  model  developed  from  data  collection  interviews  through 
verbal  protocol  (source)  techniques  and  to  determine  what  information  is  most  useful  to 
help  Expeditors  make  flightline  decisions. 

2)  To  evaluate  user  performance  and  opinions  on  enhanced  data  streams. 

The  Scientific  Study  focused  on  the  fix  or  swap  decision  construct  identified  in  the 
data  collection  and  cognitive  task  analysis  processes.  The  verbal  protocol  technique  was 
used  to  collect  data  related  to  the  types  of  information  and  processes  the  Expeditors  use 
to  make  the  fix/swap  decision.  During  the  study,  participants  were  asked  to  provide 
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estimates  of  time  to  site  for  resources.  This  estimate  was  used  as  one  dependent  variable. 
The  hypothesis  was  that  participants  would  provide  less  variable  time  to  site  estimates 
when  using  the  SSLC2  integrated  information  system  approach  than  when  using  the 
WhereNet  off-the-shelf  RFID/RTLS  technology  system,  or  current  practice  (Baseline 
condition). 

Participants  were  also  asked  to  provide  their  opinions  of  the  three  conditions.  The 
hypothesis  was  that  participants  would  prefer  the  SSLC2  approach  compared  to  the 
WhereNet  off-the-shelf  RFID/RTLS  technology  system.  Because  participants  are  so 
familiar  with  current  practice  and  often  resistant  to  change,  it  was  hypothesized  that  there 
would  be  no  difference  in  preferences  between  the  SSLC2  approach  and  the  Baseline 
current  practice  approach. 

4.2  DESIGN 

The  experimental  design  was  a  within-subject  full  factorial  design.  Each  participant 
received  three  conditions:  Baseline,  WhereNet  and  SSLC2.  The  Baseline  condition  was 
similar  to  how  Expeditors  currently  carryout  their  task.  They  were  provided  paper 
references  and  used  a  radio  for  all  communications.  The  WhereNet  condition  included 
all  the  same  paper  references  provided  in  the  Baseline  condition  as  well  as  the  WhereNet 
software  which  provided  information  regarding  the  location  of  flightline  resources 
(equipment  and  personnel).  Again,  all  communication  was  via  radio.  The  SSLC2 
condition  was  represented  through  the  simulation  system  and  included  decision  support 
as  well  as  monitoring,  location,  and  availability  of  resources.  The  radio  was  available  for 
communication,  but  mostly  used  for  receipt  of  messages. 

4.3  PARTICIPANTS 

Eighteen  people  volunteered  to  participate  in  the  study.  Two  criteria  were  established 
for  participation  in  the  study.  First,  the  participant  had  to  have  worked  as  an  Expeditor  or 
ProSuper  within  the  past  five  years.  Second,  they  must  have  performed  these  duties  for 
at  least  six  months.  Anyone  responding  to  the  study  was  permitted  to  participate  and 
provide  feedback  on  the  conditions  presented;  however,  data  were  only  analyzed 
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statistically  for  those  participants  who  met  these  two  criteria.  Sixteen  of  the  eighteen 
participants  met  these  two  criteria. 

Participants  were  from  the  445th  Airlift  Wing  AFRES  at  Wright-Patterson  AFB,  OFI; 
179th  Airlift  Wing  at  Mansfield  ANG,  OH;  180th  Fighter  Wing  at  Toledo  ANG,  OH;  and 
122nd  Fighter  Wing  at  Ft.  Wayne  ANG,  IN.  Nine  participants  dealt  with  airlift  aircraft, 
and  nine  participants  handled  primarily  fighter  aircraft.  Of  the  fighter  participants,  one 
was  not  able  to  complete  all  three  conditions,  one  was  not  able  to  complete  all  the 
questionnaires,  and  one  did  not  meet  the  criteria  established.  The  data  for  fifteen 
participants  were  analyzed  for  statistical  significance.  Nine  were  experienced  with  airlift 
aircraft  while  six  were  experienced  with  fighters. 

4.4  APPARATUS 

Three  computers  running  Windows  XP  were  used  for  the  study.  A  desktop  computer 
ran  the  SSLC2  server  and  the  Oracle  server  with  the  database.  This  machine  was  a 
Pentium  4  with  a  3  GHz  processor  and  5 1 2  megabytes  of  random  access  memory  (RAM). 
A  second  Pentium  4  desktop  computer  with  a  2.4  GHz  processor  and  1  gigabytes  of 
RAM  ran  the  WhereNet  server.  The  third  computer  was  a  laptop  that  the  participant  used 
during  the  study  to  interface  with  the  servers.  The  laptop  was  a  Pentium  4  with  a  1 .4  GHz 
processor  and  512  megabytes  of  RAM.  Internet  Explorer  was  the  browser  used  to 
communicate  with  the  servers.  The  computers  were  linked  together  using  two  hubs. 

Two  Motorola  Radius  P1225  Portable  Two  Way  Radios  were  used  for 
communication.  The  radios  operated  in  the  450-470MHz  Ultra  High  Frequency  (UHF) 
band,  and  had  two  channels  with  a  power  output  of  four  watts.  The  participant  used  the 
radio  to  contact  various  maintenance  personnel.  Maintenance  personnel  were  role  played 
by  one  of  the  researchers.  At  one  location,  the  base  communications  function  requested 
that  their  radios  be  used  to  alleviate  frequency  coordination  problems.  No  data  are 
available  on  the  types  of  radio  used  in  that  location. 

4.5  RESEARCH  TEAM 

The  study  teams  consisted  of  five  researchers  with  different  roles.  The  Lead 
Experimenter  was  responsible  for  interacting  with  the  participant.  This  person 


37 


administered  all  forms,  asked  all  question  probes,  and  generally  answered  participant 
questions.  The  Data  Collector  sat  in  the  room  with  the  Lead  Experimenter  and  the 
participant  and  was  responsible  for  capturing  all  discussions  occurring  during  the  session. 
This  included  video,  audio,  and  typewritten  information.  The  SME  was  responsible  for 
making  the  scenario  come  alive.  To  simulate  actual  flightline  conditions,  radios  were 
used  to  communicate  information  occurring  on  the  flightline.  These  radio 
communications  occurred  between  the  SME  and  the  participant.  Events  in  the  scenario 
were  scripted  ahead  of  time.  The  fourth  researcher  role  was  System  Administrator.  This 
person  set  up  the  equipment  and  was  responsible  for  advancing  and  playing  the  software 
scripts  associated  with  each  scenario.  These  software  scripts  were  synchronized  with  the 
SME  verbal  scripts  to  enhance  the  realism  of  the  events  from  the  simulated  flightline. 
For  example,  the  SME  might  call  over  the  radio  and  tell  the  participant  a  resource  was  on 
its  way;  simultaneously  the  technology  would  simulate  this  movement.  The  final 
researcher  was  the  AFRL/HEAL  Research  Coordinator  and  was  responsible  for  officially 
setting  up  the  interviews  and  interfacing  with  base  personnel.  This  person  monitored  all 
interviews  from  a  functional  perspective  and  ensured  the  experiments  complied  with  all 
Air  Force  and  base  requirements.  The  SME,  System  Administrator,  and  Research 
Coordinator  were  located  in  a  room  adjacent  to  the  room  with  the  participant,  the  Lead 
Experimenter,  and  the  Data  Collector. 

4.6  STIMULI  AND  MATERIALS 

4.6.1  Experiment  Stimuli 

The  experiment  stimuli  consisted  of  an  overall  scenario  depicting  the  deployed 
environment,  base  conditions,  and  aircraft  missions.  A  base  map  was  also  provided 
depicting  the  flightline  with  parking  locations,  aircraft  maintenance  units,  ready  line  and 
other  relevant  base  locations. 

4.6.2  Baseline  Stimuli 

Baseline  stimuli  consisted  of  paper  information  sheets  which  included  a  personnel 
roster,  equipment  list,  and  a  daily  status  sheet  that  included  each  aircraft  with  their  status, 
daily  flight  schedule,  and  maintenance  schedule. 
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4.6.3  WhereNet  Stimuli 

WhereNet  stimuli  consisted  of  the  same  paper  information  sheets  used  in  the  Baseline 
condition  as  well  as  the  WhereNet  software.  Figures  9  and  10  illustrate  the  two  primary 
visual  screens  used  in  WhereNet.  One  screen  is  a  report  view  (Figure  9)  which  allows  the 
user  to  view  resources  in  a  list  format  and  to  filter  the  information.  The  second  screen 
(Figure  10)  is  a  map  view  which  allows  the  user  to  see  the  geographic  location  of  the 
resources  on  a  map. 
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Figure  9:  WhereNet  Report  View 


39 


Figure  10:  WhereNet  Map  View 

4.6.4  SSLC2  Stimuli 

SSLC2  consisted  of  the  simulation  test  system.  During  this  condition  participants 
were  not  given  any  paper  materials.  The  system  provided  aircraft  status  and  location, 
aircraft  maintenance  schedules,  daily  flying  schedules,  personnel  and  equipment 
resources  and  their  locations,  tasks  necessary  to  fix  the  broken  aircraft,  tasks  necessary  to 
swap  an  aircraft,  time  to  complete  fix  tasks,  time  to  get  resources  to  the  fix  site,  resource 
types  necessary  to  complete  tasks,  specific  resources  assigned  to  tasks,  and  a 
recommendation  to  fix  or  swap  an  aircraft.  Figure  1 1  illustrates  the  geographic  view  of 
SSLC2.  Figure  12  illustrates  the  schedule  view. 
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4.6.5  Scenarios 

The  study  was  conducted  by  creating  scenarios  in  which  an  aircraft  on  the  flight 
schedule  has  a  specific  maintenance  problem.  The  participant  acting  as  Expeditor  must 
analyze  the  problem  and  determine  whether  to  fix  or  swap  the  aircraft.  The  scenarios 
were  designed  to  have  three  triggers.  Time  would  advance  between  each  trigger.  At  the 
beginning  of  the  scenario,  the  participant  would  monitor  the  flightline  (review  the  current 
state  of  the  aircraft  and  flightline).  The  first  trigger  was  an  aircraft  maintenance  problem. 
The  problem  has  already  been  assessed  and  verified  by  a  specialist.  Once  the  information 
is  received  the  participant  works  the  problem,  decides  to  fix  or  swap,  provides  estimates 
to  time,  and  begins  the  fix.  After  completing  this  trigger,  a  second  and  third  trigger 
occurred  in  which  a  resource  had  to  be  replaced.  One  trigger  was  a  personnel  resource 
and  one  was  an  equipment  resource.  The  participant  would  work  through  replacing  the 
resource. 

Four  scenarios  were  created,  each  with  a  different  aircraft  problem.  Scenarios  were 
adapted  for  either  fighter  or  airlift  participants.  The  scenarios  included  the  following:  1) 
Gland  Nut  Change,  2)  Cockpit  Glass  Change  (airlift  version)  and  Canopy  Glass  Change 
(fighter  version),  3)  Fuel  Leak  and  4)  Prop  Change  (airlift  version)  and  Engine 
Compressor  Stall  (fighter  version).  Three  of  the  scenarios  were  used  for  the  three  study 
conditions  while  the  fourth,  the  Prop  Change/Engine  Compressor  Stall  scenario,  was  used 
for  training.  All  scenarios  are  presented  in  Appendix  C. 

4.6.6  Questionnaires  and  Surveys 

Participants  were  given  questionnaires  to  complete  throughout  the  evaluation  process. 
They  were  given  a  pre-test  questionnaire,  a  post-condition  questionnaire  after  completing 
each  of  the  three  experimental  conditions,  and  a  three-part  post-test  questionnaire  at  the 
end  of  the  study. 

The  pre-test  questionnaire  requested  information  such  as  the  participant’s  years  of 
experience  on  the  aircraft,  their  experience  in  using  computing  systems,  and  two 
questions  related  to  the  criteria  for  acceptance  (i.e.,  Expeditor  or  ProSuper  experience 
within  the  past  five  years  and  experience  for  a  minimum  of  six  months). 
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The  post-condition  questionnaire  consisted  of  fifteen  statements  using  a  seven-point 
rating  scale  format  where  a  rating  of  1  indicated  “strongly  disagree”  and  7  indicated 
“strongly  agree”  with  the  statement.  Three  of  the  questions  related  specifically  to 
comparing  the  value  of  the  technologies  in  supporting  human  performance  [15].  Three 
questions  were  targeted  toward  estimating  the  technologies  ability  to  support  situation 
awareness  and  decision-making. 

Part  1  of  the  post-test  questionnaire  asked  participants  to  rank  order  their  preference 
for  technologies  along  five  separate  dimensions.  The  first  three  rankings  were  aimed  at 
the  monitoring,  locating,  and  identifying  availability  of  resources.  The  fourth  and  fifth 
rankings  looked  specifically  at  tracking  time  associated  with  resources  and  making 
decisions  about  fixing  or  swapping  the  aircraft.  Part  2  and  3  of  the  post-test  questionnaire 
were  aimed  at  collecting  additional  information  relevant  to  further  development  of 
SSLC2  capabilities. 

4.7  PROCEDURE 

The  study  was  conducted  in  a  standard  office  or  conference  room  at  the  various  study 
locations.  Each  participant  completed  the  entire  study  in  one  3  lA  hour  (approximate) 
session.  After  informal  introductions,  they  were  briefed  about  the  purpose  of  the  study 
and  were  given  an  informed  consent  form  to  sign  followed  by  the  pre-test  questionnaire, 
study  conditions,  and  post-test  questionnaire.  The  study  conditions  were  counter¬ 
balanced  across  eighteen  participants.  The  three  scenarios  where  then  randomly  assigned 
to  the  conditions  without  replacement.  Across  the  eighteen  participants  each  scenario 
was  paired  with  each  experimental  condition  a  total  of  six  times.  Participants  were  then 
randomly  assigned  to  conditions. 

Each  condition  started  with  a  training  session.  The  training  session  consisted  of  two 
parts.  The  radio  communication  system  was  briefly  explained,  including  the  call  signs. 
All  participants  were  familiar  with  how  to  use  the  radio.  For  the  SSLC2  and  WhereNet 
conditions  the  participant  was  trained  on  how  to  use  the  software.  After  technology 
training  the  participant  was  given  time  to  interact  with  the  technology  until  they  felt 
comfortable.  The  second  part  of  training  allowed  the  user  to  step  through  a  sample 
scenario.  During  scenario  training  two  triggers  were  used,  the  first  aircraft  problem 
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trigger  and  one  resource  trigger.  During  this  part  of  training  the  participant  used  the 
technology  provided,  and  interacted  with  the  SME  over  the  radio  to  work  through  the 
problem.  They  practiced  the  verbal  protocol  thinking  aloud  technique.  After  training, 
subjects  were  asked  to  rate  their  comfort  with  the  technology  on  a  scale  of  1-7,  with 
seven  being  the  most  comfortable  and  then  participated  in  the  study  trial. 

During  the  study  the  participant  was  encouraged  to  think  aloud  as  they  were  making 
decisions.  Additionally,  the  Lead  Experimenter  asked  question  probes  at  appropriate 
times  to  capture  information  such  as  comfort  with  the  technology,  strategies  used  for 
making  decisions,  and  estimates  of  time  associated  with  movement  on  the  flightline. 
After  each  condition  was  complete,  the  participant  filled  out  the  post-condition 
questionnaire.  This  process  was  iterated  for  each  of  the  three  conditions.  Once  all  three 
conditions  were  complete  the  participant  was  given  the  post-test  questionnaire  and 
thanked  for  their  participation. 

5  Results  and  Discussion 

This  section  begins  by  analyzing  and  discussing  specific  results  including 
presentation  of  participant  demographics,  analysis  of  participant  ratings  and  rankings, 
statistical  analysis  of  the  dependent  variables  comfort  ratings,  time  to  fix,  and  time  to  site, 
and  verbal  protocol  analysis.  A  summary  of  the  results  with  respect  to  the  study  of 
objectives  is  presented,  followed  by  a  discussion  of  observations  which  summarizes 
feedback  and  suggestions. 

5.1  PARTICIPANT  DEMOGRAPHICS  INFORMATION 

Participants  represented  both  the  Fighter  and  Airlift  domains.  The  nine  Fighter 
participants  had  an  average  of  18.85  years  experience  while  those  working  Airlift  had  an 
average  of  18.78  years  of  experience.  Four  participants  had  experience  working  both 
Fighters  and  Airlift.  Various  maintenance  supervisory  roles  were  represented.  Table  3 
lists  the  flightline  roles  and  frequency.  Participants  had  on  average  4.6  years  of 
experience  being  an  Expeditor.  Two  participants  had  experience  as  ProSupers  but  were 
not  Expeditors.  There  was  an  average  of  2.9  years  of  ProSuper  experience.  Three 
participants  had  no  ProSuper  experience. 
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Table  3:  Field  Test  Subject  Roles 


Role 

Frequency 

Aircraft  Maintenance  Superintendent 

2 

Aircraft  Maintenance  Supervisor 

1 

Aircraft  Mechanic  Supervisor 

3 

Avionics  Tech 

1 

Avionics  Tech/Expeditor 

1 

Deputy  Maintenance  Superintendent 

1 

Expeditor 

3 

Expeditor/Pro  Super 

1 

Expeditor/Maintenance  Superintendent 

1 

ProSuper 

4 

All  but  one  participant  said  they  use  the  computer  daily.  The  exception  used  the 
computer  more  than  once  a  week.  All  participants  had  been  deployed  overseas,  with  an 
average  of  19.7  times. 


5.2  RATINGS  AND  RANKINGS  ANALYSIS 

The  post-condition  questionnaires  and  post-test  questionnaires  were  analyzed  for 
fifteen  of  the  eighteen  participants  who  met  the  screening  criteria  and  completed  the 
entire  study.  The  results  for  each  type  of  questionnaire  are  described  in  the  sections 
below  followed  by  a  brief  summary. 

This  section  concludes  with  two  further  analyses,  a  Post-Study  Questionnaire  Part  2 
which  addressed  Decision  Impacts,  and  a  Post-Study  Questionnaire  Part  3  which 
addressed  Data  Elements. 

Additional  findings  from  rating  and  ranking  analyses  regarding  Key  Performance 
Parameters  are  located  in  Section  5.8  SSLC2  Program  Key  Performance  Parameters. 
Findings  related  to  these  parameters  include  rating  and  ranking  data  for  tracking  time, 
assessing  human  performance,  providing  situation  awareness,  assisting  with  decision 
making  using  the  three  technologies. 

5.2.1  Post-Condition  Questionnaire  Results 

The  post-condition  questionnaire  asked  participants  to  rate  their  agreement  with 

fifteen  statements  on  scale  from  1  to  7  (1  strongly  disagree,  4  neutral,  7  strongly  agree) 
(see  Table  4).  An  overall  analysis  averaging  ratings  across  all  fifteen  statements  was 
conducted.  Additionally,  each  individual  statement  was  evaluated  across  the  three 
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conditions.  In  all  cases,  an  Analysis  of  Variance  (ANOVA)  with  a  criterion  level  of  0.05 
was  conducted  with  a  Greenhouse-Geisser  correction.  (Note  that  Greenhouse-Geisser  is  a 
very  conservative  correction.)  Paired  comparisons  were  also  conducted  when  effects 
were  significant  using  Tukey’s  test.  Table  4  shows  the  statistical  results  and  mean  ratings 
for  each  condition.  Data  for  one  participant  was  dropped  due  to  missing  data  for  several 
questions. 


Table  4 :  Mean  Ratings  with  Tukey  Groupings* 


Statement  (1  -  Strongly  Disagree,  7  -  Strongly  Agree):  F  values,  p,  and  Condition  Means 

SS=  Smart  Systems,  WN  =  WhereNet,  BL  =  Baseline,  Bold  p  indicates  significant  findings  at  the  .05 
level 

1 .  There  was  enough  information  at  all  times  to 
carry  out  my  tasks  and  make  decisions. 

F  (1,13)  =  1.28,  fi=  .2783 

SS=6.43  WN=5.86  BL=5.64 

9.  The  technology  improved  my  resource  utilization 
because  1  know  where  all  my  resources  were. 
F(1,13)=11.41,  p  =.0049 

SS=6.57  (A)  WN=5.93  (A)  BL=4.62  (B) 

2.  There  was  enough  information  to  monitor  the 
status  of  resources  on  the  flightline. 

F(1,13)=7.68,  fi=.0159 

SS=6.78  (A)  WN=6.07  (A,  B)  BL=5.21  (B) 

10.  The  technologies  provided  were  usable. 

F(1 ,13)=3.54,  p  =.0825 

SS=6.50  WN=6.07  BL=5.72 

3. 1  was  able  to  determine  where  my  equipment 
resources  were. 

F(1,13)=6.21,e  =  0270 

SS=6.64  (A)  WN=6.57  (A)  BL=5.00  (B) 

11.1  was  provided  with  useful  decision  support. 

F(1 ,13)=2.05,  p  =.1758 

SS=6.36  WN=6.21  BL=5.50 

4. 1  was  able  to  determine  where  my  personnel 
resources  were. 

F(1,  13)=9.39,  p  =.0090 

SS=6.14  (A)  WN=6.64  (A)  BL=4.57  (B) 

12.  Decision  support  as  provided  through  this 
technology  would  positively  impact  flightline 
maintenance.  F(1,13)=  2.71,  p  =.1236 

SS=6.21  WN=6.21  BL=5.35 

5.  1  was  able  to  identify  the  availability  of  my 
equipment  resources. 

F=(1,13)=6.44,  p=0248 

SS=6.43  (A)  WN=6. 28(A)  BL=4.78  (B) 

13.  It  was  easy  to  notice  when  there  were 
problems  with  aircraft. 

F(1,13)=3.15,  p  =.0993 

SS=6.14  WN=4.71  BL=5.00 

6.  1  was  able  to  identify  the  availability  of  my 
personnel  resources. 

F(1 ,  13)=4.39,  p  =.0561 

SS=6.07  WN=6.07  BL=4.64 

14.  When  problems  occurred,  it  was  easy  to 
understand  the  status  and  location  of  resources 
needed  to  address  the  problem. 

F(1 ,13)=3.03,  p  =.1053 

SS=6.57  WN=6.21  BL=5.57 

7. 1  was  able  to  determine  what  specific  tasks 
needed  to  be  accomplished. 

F(1,13)=1.028,e=.3291 

SS=6.43  WN=6.07  BL=5.92 

15.  When  problems  occurred,  1  was  able  to  predict 
whether  I  could  meet  the  flying  schedule. 
F(1,13)=5.06,JJ  =  .0424 

SS=6.00  (A)  WN=5.64  (A,  B)  BL=4.71  (B) 

8. 1  was  able  to  track  time  associated  with  the 
movement  of  resources  on  the  flightline. 
F(1,13)=5.78,  p  =  .0318 

SS=6.36  (A)  WN=5.29  (A,  B)  BL=4.79  (B) 

Collapsing  across  all  15  statements  combined: 

F(1 ,13)  =  8.98,  p  =  .0103 

SS=6.37  (A)  WN=5.99  (A)  BL=5.13  (B) 

*  Letters  next  to  means  indicate  Tukey  Groupings.  Means  with  the  same  letter  are  not 
significantly  different. 


46 


When  collapsing  across  all  15  agreement  statements  (F  (1,13)  =8.98,  p  =.0103),  there 
was  a  statistically  significant  difference  in  ratings  between  Smart  Systems  and  Baseline, 
and  WhereNet  and  Baseline.  Examining  each  statement  separately  (Table  4),  8  of  the  15 
statements  showed  no  significant  differences  among  conditions.  Of  the  7  statements  that 
showed  statistical  significance.  Smart  Systems  was  rated  higher  than  Baseline.  Four  of 
the  7  statements  indicated  WhereNet  was  rated  higher  than  Baseline.  Smart  Systems  was 
not  rated  higher  than  WhereNet  for  any  of  the  7  statements.  See  Figure  13  for  mean 
rating  scores  collapsed  across  all  fifteen  rating  statements. 

Statements  13-15  all  dealt  with  aspects  related  to  situation  awareness  (noticing, 
understanding,  and  predicting)  and  were  also  collapsed  and  analyzed  for  overall 
situational  awareness  assessment  (F(l  ,1 3)  =  4.96,  p=.0442).  Results  indicated  that  Smart 
Systems  was  rated  significantly  higher  than  Baseline.  However,  when  looking  at  each  of 
these  questions  individually  we  see  that  only  one  aspect  reaches  statistical  significance  - 
predicting.  Statements  related  to  noticing  and  understanding  showed  no  difference  among 
the  ratings.  We  can  not  conclude,  therefore,  that  situation  awareness  is  improved  for  all 
aspects  of  Situational  awareness. 

Statements  10-12  dealt  with  human  performance  with  the  tools  (usability,  usefulness, 
and  impact  to  decision  making),  these  statements  were  collapsed  to  analyze  for  overall 
human  performance  with  the  tool.  Results  were  not  statistically  significant. 
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Figure  13:  Mean  Rating  Scores  Collapsed  Across  All  15  Rating  Statements 

5.2.2  Post-Study  Questionnaire  Part  1:  Ranking  Results 

Part  1  of  the  post-study  questionnaires  asked  participants  to  rank  order  the  three 

conditions  for  five  purposes:  monitoring  resources,  locating  resources,  identifying 
resource  availability;  tracking  time  associated  with  resources,  and  making  the  fix/swap 
decision.  The  rankings  for  all  five  questions  were  collapsed  and  analyzed  for  differences. 
Also,  the  mean  rankings  for  each  ranking  question  were  evaluated  across  the  three 
conditions.  In  all  cases,  an  ANOVA  with  a  criterion  level  of  0.05  was  conducted  with  a 
Greenhouse-Geisser  correction.  Paired  comparisons  were  also  conducted  when  effects 
were  significant  using  Tukey’s  test.  Results  are  presented  in  Table  5. 

When  collapsing  across  the  five  statements  there  was  a  statistically  significant 
difference  between  Smart  Systems  and  WhereNet  and  between  Smart  Systems  and 
Baseline  (see  Table  5  for  mean  rankings).  There  were  no  statistical  differences  between 
WhereNet  and  Baseline.  Examining  results  for  each  ranking  separately  shows  differences 
among  the  three  conditions  varied  based  on  the  ranking  statement.  There  was  a 
statistically  significant  difference  between  Smart  Systems  and  WhereNet  for  4  of  the  5 
statements  (Monitoring,  Locating,  Tracking  time  and  Fix/Swap)  with  Smart  Systems 
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being  ranked  higher.  There  was  a  significant  difference  between  Smart  Systems  and 
Baseline  for  3  of  the  5  statements  (Monitoring,  Locating  and  Identifying)  with  Smart 
Systems  being  ranked  higher.  Ranking  was  significantly  different  between  WhereNet 
and  Baseline  only  when  making  the  Fix/Swap  decision.  In  this  case  Baseline  was  ranked 
higher.  These  results  suggest  that  there  was  little  difference  between  Baseline  and 
WhereNet  conditions  in  terms  of  preference.  Figure  14  shows  the  percentage  that  each 
condition  was  ranked  either  first,  second  or  third  across  all  five  questions  (225  rankings 
total).  SSLC2  had  48%  of  the  first  place  rankings,  WhereNet  had  only  8%  of  first  place 
rankings,  and  Baseline  had  28%.  WhereNet  had  54%  of  the  third  place  rankings  and 
SSLC2  had  only  5%  of  third  place  rankings. 


Table  5:  Mean  Rankings  and  Tukey  Groupings* 


Preference  Rating:  1=first  Preference,  3=  third  preference 

Ranking  Statement  and  F  test 

Means  and  Tukey  Groupings 

Monitoring  Status  of  Resources 

F(l, 14)  =  8.34,11  =  .0119 

SS=  1.33  (A)  WN  =  2.53  (B)BL  =  2.13(B) 

Locating  Resources 

F(1 ,14)  =  5.44,  e  =  - 0351 

SS  =  1 .40  (A)  WN  =  2.20  (B)  BL  =  2.40  (B) 

Identifying  Resource  Availability 

F(1, 14)  =  3.90,  fi  =  .0683 

SS  =  1.47  (A)  WN  =  2.20  (A,  B)  BL  =2.33  (B) 

Tracking  Time  Associated  with  Resources 
F(1,14)  =  11.20,  e=  .0048 

SS  =  1 .33  (A)  WN  =  2.67  (B)  BL  =  2.00  (A,  B) 

Making  Fix/Swap  Decision 

F(1, 14)  =  9.86,  fi  =  . 0072 

SS  =  1.53  (A)  WN  =  2.73  (B)  BL  =  1.73  (A) 

Collapsing  across  all  Ratings 

F(1, 14)  =  9.52,  £  =  .0080 

SS=  1.41  (A)  WN  =  2.47(B)  BL  =  2.12(B) 

*  Letters  next  to  means  indicate  Tukey  Groupings.  Means  with  the  same  letter  are  not 
significantly  different. 
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1st  Choice  2nd  Choice  3rd  Choice 

Figure  14:  Percentages  that  each  condition  was  ranked  1st,  2nd,  and  3rd  choice 

5.2.3  Summary  of  Ratings  and  Ranking  Results 

Table  6  summarizes  the  results  related  to  five  types  of  tasks.  Results  from  the 

rankings  indicated  that  SSLC2  was  preferred  to  WhereNet  and  Baseline.  When  ranking 
the  three  conditions,  there  was  only  one  difference  between  Baseline  and  WhereNet 
conditions.  Baseline  was  ranked  higher  than  WhereNet  for  making  the  Fix/Swap 
decision.  However,  WhereNet  was  rated  higher  than  Baseline  in  four  of  the  15  rating 
statements  (Table  4).  These  statements  were  related  to  location  of  equipment  and 
personnel,  availability  of  equipment,  and  improved  resource  utilization.  It  should  be 
noted  that  WhereNet  only  provides  information  about  location,  not  availability  of 
resources.  However  location  is  sometimes  an  indication  of  availability,  e.g.,  if  personnel 
are  located  in  the  AMU  the  expeditor  may  assume  they  are  available.  For  rating 
statements  there  were  no  differences  between  Smart  Systems  and  WhereNet,  yet  Smart 
Systems  was  ranked  higher  in  terms  of  preference.  Users  commented  that  they  felt 
interacting  with  WhereNet  was  somewhat  difficult;  however  they  did  like  the  ability  to 
filter  and  sort  resources. 
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Table  6:  Summary  of  differences  among  conditions 


Task 

Rankings 

Ratings 

Monitoring  Resources 

SSLC2  was  preferred  over 
WhereNet  and  Baseline 

SSLC2  was  rated  higher  than 
Baseline 

Locating  Resources 

SSLC2  was  preferred  over 
WhereNet  and  Baseline 

SSLC2  and  Wherenet  were 
rated  higher  than  Baseline 

Resource  Availability 

SSLC2  was  preferred  over 
Baseline 

SSLC2  and  WhereNet  were 
rated  higher  than  Baseline 

Tracking  Time 

SSLC2  was  preferred  over 
WhereNet 

SSLC2  were  rated  higher 
than  Baseline 

Resource  Utilization  based 
on  knowledge  of  location 

Not  Applicable 

WhereNet  was  rated  higher 
than  Baseline 

Fix/Swap  Decision 

SSLC2  and  Baseline  were 
preferred  over  WhereNet 

No  difference  for  decision 
support 

5.2.4  Post-Study  Questionnaire  Part  2:  Decision  Impacts 

Post  study  questionnaire  Part  2  requested  participants  to  assign  a  rating  indicating  the 

importance  of  certain  information  on  decisions.  In  the  first  half  of  the  questionnaire 
participants  rated  7  information  elements  (1-not  at  all  important  to  7-extremely 
important)  on  their  fix/swap  decision.  The  frequency  data  for  this  question  is  located  in 
Section  3.1.1  Appendix  B.  Table  7  presents  the  average  ratings  for  each  data  element. 

Table  7:  Average  Rating  for  Information  IMPORTANCE  WHEN  Making  the  Fix 
or  Swap  Decision 


Data  Element 

Average  Rating  (1-7) 

Today’s  flying  schedule 

7.00 

Tomorrow’s  flying  schedule 

6.00 

Weekly  flying  schedule 

5.00 

Monthly  flying  schedule 

4.07 

Flying  schedule  effectiveness  metric  (FSE) 

4.36 

Aircraft  availability  metric  (AA) 

6.00 

Mission  capable  (MC)  rate 

5.07 

Over  half  of  the  participants  said  the  Aircraft  Availability  (AA)  Metric  was 
“extremely  important”  to  the  fix/swap  decision.  All  of  the  participants  said  Today’s 
Flying  Schedule  was  “extremely  important”  when  making  the  fix  or  swap  decision.  Other 
information  that  was  “somewhat  important”  was  Tomorrow’s  Flying  Schedule  and  the 
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Weekly  Flying  Schedule.  Additional  data  points  were  Flying  Schedule  Effectiveness 
(FSE)  Rate,  Monthly  flying  Schedule,  and  MC  Rate  which  mainly  fell  into  the  “slightly 
important”  or  “neutral  categories.” 

The  second  part  of  the  questionnaire  asked  participants  to  rate  how  various  decisions 
they  make  impact  FSE,  AA,  and  MC  rates.  Table  8  provides  a  summary  of  the  average 
ratings.  Frequency  information  can  be  found  in  Section  3.1.1  Appendix  B. 


Table  8:  Average  Rating  for  How  Decisions  Impact  FSE,  AA  and  MC  Rates 


Data  Element 

Average  Rating  (1-7) 

What  Personnel  should  be  assigned  to  maintenance  tasks? 

5.86 

Fix  or  Swap? 

6.36 

Should  the  flight  be  delayed? 

6.36 

Should  we  CANN? 

6.00 

Which  aircraft  should  be  assigned  to  which  sortie? 

6.21 

How  should  aircraft  be  assigned  for  the  week? 

5.36 

How  will  aircraft  be  assigned  for  the  month? 

4.46 

How  should  today's  flying  schedule  be  changed? 

6.15 

The  fix/swap  decision  was  ranked  highest  with  nine  respondents  saying  it  was 
“extremely  important”  in  impacting  metrics.  Additional  “extremely  important”  decisions 
were:  How  should  today’s  flying  schedule  be  changed;  should  the  flight  be  delayed;  and 
which  aircraft  should  be  assigned  to  which  sorties?  Most  participants  rated  the  Monthly 
schedules  as  slightly  important  (seven  of  thirteen)  or  neutral  (four  of  thirteen).  These 
results  are  important  for  planning  what  information  and  decisions  a  future  decision 
support  tool  should  focus  on. 

5.2.5  Post-Study  Questionnaire  Part  3:  Data  Elements 

This  part  of  the  questionnaire  asked  participants  to  rate  on  a  scale  of  1  to  7,  the 

importance  of  data  elements  that  would  be  available  in  an  electronic  decision  support 
tool.  The  data  elements  were  broken  into  five  categories:  Aircraft,  Personnel,  AGE, 
Supply,  and  Other.  They  were  also  asked  to  write  in  any  data  elements  they  considered 
important.  Table  9  provides  the  average  ratings  for  each  data  element.  (See  Section  3.1.2 
in  Appendix  B  for  frequency  information.) 
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Table  9:  average  Important  Rating  for  Data  Elements  in  a  Decision  Support 
Tool 


DATA  ELEMENT 

Average  Rating  (1  -7) 

AIRCRAFT 

Location  of  aircraft 

6.64 

Current  status  of  aircraft  (FMC,  NMC,  PMC) 

7.00 

781  Discrepancies 

6.5 

CAMS  380  list  of  jobs 

5.36 

Aircraft  hours 

4.07 

Engine  hours  and  cycles 

3.92 

Time  since  last  Phase/ISO  inspection 

4.5 

Aircraft  configuration 

6.75 

Aircraft  maintenance  history 

4.57 

Monthly  aircraft  scheduled  maintenance 

5.21 

Weekly  aircraft  scheduled  maintenance 

5.86 

Weekly  flying  schedule 

6.00 

Monthly  flying  schedule 

4.71 

PERSONNEL 

Location  of  personnel  resources 

6.69 

Weekly  personnel  schedules 

5.5 

Monthly  personnel  schedules 

4.43 

Personnel  qualifications  Rank 

5.78 

Personnel  qualifications  Specialty 

6.43 

Personnel  qualifications  Skill  level  (3,5,7) 

6.71 

Personnel  qualifications  Certifications 

6.64 

Special  certification  rosters  (RedX,  ER,  intake, 
etc) 

6.86 

Personnel  status 

6.46 

AGE 

Location  of  AGE  and  other  equipment 

6.00 

Status  of  AGE  equipment  (FMC,  NMC) 

6.14 

Fuel  level  on  powered  equipment 

5.29 

Level  of  liquid  oxygen 

6.36 

Battery  level  on  equipment 

4.64 

107  requests 

3.91 

SUPPLY 

Location  of  supplies 

6.00 

Bench  Stock 

5.86 

Status  of  supplies 

6.14 

Location  of  MICAP  supply 

6.29 

Status  of  MICAP  supply 

6.38 

350  tag 

4.42 

OTHER 

Weather 

6.57 

Squadron  unit  monthly  analysis  indicators 
report 

4.33 

Under  the  category  of  Aircraft,  participants  indicated  781  Discrepancies,  Aircraft 
Configuration,  Current  Aircraft  Status  and  Aircraft  Location  were  “extremely  important” 
while  CAMS  380  Job  Listings,  Monthly  Flying  Schedule,  Weekly  Scheduled 
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Maintenance  and  Weekly  Flying  Schedules  were  “slightly  important”  or  “somewhat 
important.”  These  data  points  are  consistent  with  participant  comments  during  the  study. 
Aircraft  information  rated  “slightly  important,”  “somewhat  important”  or  “not  important 
at  all”  were  Aircraft  Maintenance  History,  Engine  Hours  and  Cycles,  Monthly  Scheduled 
Maintenance,  and  Time  since  last  Phase/Inspection  Operations  (ISO)  Inspection. 

AGE  related  data  elements  including  Fuel  Level,  Liquid  Oxygen  Level,  AGE  and 
Equipment  Location  and  AGE  Status  were  “somewhat  important”  and  “extremely 
important”  while  107  Request  and  Battery  Levels  were  not  overwhelmingly  important  to 
the  participants. 

Participants  ranked  personnel  data  elements  associated  with  location,  schedule,  and 
qualifications.  Participants  overwhelmingly  ranked  personnel  location,  specialties, 
certifications,  skill  levels  and  rosters  as  “extremely  important”  information  points  for 
doing  their  jobs.  This  data  element  grouping  was  rated  the  highest  compared  to  AGE  or 
supply  information.  This  data  reinforces  many  comments  during  the  field  test  relating  to 
difficulty  in  finding  the  right  people  at  the  right  time. 

Participants  frequently  asked  questions  about  supply  during  the  study  and  their  data 
rankings  indicated  Mission  Capability  (MICAP)  and  Supply  Status  were  “extremely 
important.”  MICAP  and  Supply  Location  and  Bench  Stock  were  rated  “somewhat 
important.” 

5.3  COMFORT  LEVEL 

Before  beginning  each  condition,  immediately  following  practice,  the  participants, 
were  asked  to  rate  their  comfort  level  with  the  technology  on  a  scale  of  1-7,  with  7  being 
the  most  comfortable.  The  comfort  levels  were  averaged  across  participants  within  both 
groups.  Fighter  and  Airlift.  A  t-test  was  conducted  across  all  three  conditions,  collapsing 
across  aircraft  type.  Data  were  also  analyzed  by  aircraft  type  using  a  t-test.  Significance 
criterion  was  set  at  0.05.  Figure  15  illustrates  the  data. 
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Figure  15:  Mean  Comfort  Rating 

There  was  a  statistically  significant  difference  among  all  three  conditions  t  (43)  = 
2.021,  (p  <  0.05).  As  expected  most  participants  provided  the  highest  ratings  (were  most 
comfortable)  in  the  baseline  condition  using  a  radio.  Participants  were  more  comfortable 
with  SSLC2  after  initial  training  than  with  WhereNet.  Analyzing  the  data  by  aircraft 
type,  there  was  a  statistically  significant  different  between  WhereNet  and  Baseline 
comfort  levels  for  the  Fighter  participants  ( t  (22)  =  2.093,  (p  <  0.05).  For  Airlift 
participants,  there  was  a  statistically  significant  difference  among  all  three  comfort  levels. 

5.4  TIME  TO  FIX 

During  the  experiment  participants  were  asked  to  provide  an  estimate  of  the  time  to 
fix  the  problem  aircraft.  This  estimate  was  provided  during  the  first  portion  of  the 
scenarios  when  making  the  fix/swap  decision.  Not  all  participants  provided  an  estimate. 
Out  of  45  possible  data  points,  eight  were  missing.  For  the  remaining  data  points,  the 
mean  time  to  fix  estimate  was  determined  for  the  three  conditions.  The  mean  time  in 
minutes  were  Baseline  X  =  343.85;  WhereNet  X  =  354.54;  SSCL2  X  =  440.00.  A  t-test 
was  conducted  to  exam  the  differences  between  means.  There  was  a  statistically 
significant  difference  between  the  SSCL2  and  Baseline  Conditions  t  (37)  =  2.034,  (p  < 
0.05).  There  were  no  other  significant  differences.  SSLC2  provided  time  to  fix  values 
for  the  participant.  For  the  Baseline  and  WhereNet  conditions,  participants  had  to 
compute  the  information  or  they  specifically  requested  it  from  an  outside  source  such  as 
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the  specialist  over  the  radio.  The  variance  around  the  means  for  the  Baseline  and 
WhereNet  conditions  is  quite  high  (Std  Dev  =  130  for  Baseline  and  135  for  WhereNet) 
compared  to  SSLC2  as  would  be  expected  (Std  Dev  =  62).  When  participants  had  to 
estimate  fix  times,  the  times  were  shorter  than  those  specified  in  SSLC2.  It  should  be 
noted  that  SSCL2  fix  times  were  determined  by  the  SMEs  and  AFRL/HEAL  Functionals. 

5.5  TIME  TO  SITE 

During  each  scenario  participants  made  decisions  about  resources,  both  equipment 
and  personnel.  They  had  to  choose  and  locate  a  resource  and  were  asked  to  give  an 
estimate  of  the  time  it  would  take  for  the  resource  to  reach  the  problem  site  (time  to  site). 

There  were  a  total  of  69  data  points  available  for  this  analysis,  with  23  data  points  in 
each  condition.  The  mean  time  to  site  estimates  in  minutes  were:  Baseline  X  =  6.52; 
WhereNet  X  =  6.74;  and  SSLC2  X  =  3.89.  A  t-test  was  conducted  to  determine 
differences  between  the  three  conditions. 

There  was  a  statistically  significant  difference  between  the  SSLC2  and  WhereNet 
Conditions  /  (69)  =  1.996,  (jj  <  0.05).  There  were  no  other  significant  differences.  The 
variance  related  to  estimates  were  greater  in  the  Baseline  and  WhereNet  conditions  (Std 
Dev  Baseline  =  4.48;  Std  Dev  WhereNet  =  4.50)  compared  to  SSLC2  (Std  Dev  =  2.18). 
SSLC2  calculated  time  to  site  with  a  straight  line  estimate.  Participant’s  estimates  were 
primarily  based  on  their  experience  that  it  should  not  take  more  than  5  to  10  minutes  for 
someone  or  something  to  arrive  once  requested.  Most  participants  indicated  verbally  that 
the  time  for  resources  to  arrive  would  not  change  their  Estimated  Time  to  Completion 
(ETIC)  or  time  to  fix  estimates. 

5.6  VERBAL  PROTOCOL  ANALYSIS 

Participants  were  asked  to  think  aloud  during  each  scenario.  Their  verbal  statements 
were  recorded  and  notes  were  also  taken  in  real  time.  The  verbal  information  for  fifteen 
subjects  was  evaluated,  across  all  three  conditions  resulting  in  analysis  of  45  scenarios. 
(Note  that  data  analysis  of  questionnaires  had  data  for  fourteen  participants  because  one 
participant  did  not  complete  all  questionnaires.  However,  the  participant  did  complete  all 
scenarios  so  the  data  were  included  in  this  analysis.)  The  data  were  analyzed  in  several 
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different  ways.  First  the  information  was  broken  into  steps.  For  each  step,  the  verbal 
information  was  dissected  into  six  categories.  Section  3.2  of  Appendix  B,  Data  Collection 
illustrates  the  data  table  used.  The  first  category  is  Information.  This  category  refers  to 
the  type  of  information  the  participant  was  seeking,  distributing  or  working  with  during 
the  step.  The  second  category  is  Source,  which  refers  to  where  they  were  seeking  the 
information.  The  third  category  is  Destination,  referring  to  where  the  participant  sent  or 
distributed  the  information.  The  fourth  category  is  Decision  which  indicates  what 
decision  they  were  working  on.  The  fifth  category  is  Time,  which  was  used  to  input  the 
time  to  site  or  time  to  fix  information  that  participant  provided  during  the  scenario.  The 
sixth  category  is  Process  which  is  a  description  of  steps  or  the  process  the  participants 
were  taking.  Data  from  each  of  these  steps  was  analyzed  as  detailed  below. 

5.6.1  Information 

The  types  of  information  the  participants  used  during  the  task  were  determined  and  a 
frequency  count  was  tabulated  showing  how  often  the  information  was  referred  to  or  used 
across  all  fifteen  subjects  and  three  conditions  (45  scenarios).  Section  3.2.1  in  Appendix 
B,  Data  Collection  provides  the  frequency  count  data  and  categories  of  information.  The 
data  helps  to  determine  the  type  of  information  the  participants  needed  to  complete  the 
tasks.  Because  participants  were  specifically  instructed  to  provide  time  to  site  and  time 
to  fix  information  as  well  as  find  equipment  and  personnel  resources  these  types  of 
information  have  the  highest  frequency  counts.  If  these  data  were  not  specifically 
requested,  they  may  not  have  referred  to  them  as  often. 

5.6.1. 1  Equipment  Resources 

Equipment  was  broken  into  four  categories.  1)  Equipment  allocation  -  participants 
allocated  or  requested  a  piece  of  equipment.  2)  Equipment  availability,  3)  Equipment 
location,  and  4)  Equipment  movement.  The  frequency  counts  are  summarized  in  Table  10 
below. 
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Table  10:  Equipment  Information  Analysis 


Information 

Baseline 

SSLC2 

Total 

Equipment  Allocation 

15 

15 

2 

32 

Equipment  Availability 

10 

6 

2 

18 

Equipment  Location 

4 

24 

15 

43 

Equipment  Movement 

0 

1 

0 

1 

For  the  categories  of  equipment  allocation  and  availability  there  are  similar  frequency 
counts  between  Baseline  and  WhereNet  conditions,  which  are  both  higher  than  SSLC2. 
SSLC2  provides  allocation  automatically  and  selects  equipment  based  on  location  and 
availability.  Therefore,  SSLC2  reduced  the  number  of  times  Expeditors  had  to  request  or 
find  equipment.  In  terms  of  equipment  location,  participants  in  the  Baseline  condition  did 
not  refer  to  equipment  location.  In  WhereNet  and  SSLC2  they  used  the  computer  systems 
to  search  for  specific  equipment.  WhereNet  had  the  largest  frequency  count  for  this 
information.  In  all  scenarios  participants  were  required  to  replace  a  piece  of  equipment 
that  had  malfunctioned  or  gone  missing.  In  the  Baseline  condition  they  called  AGE  and 
requested  the  equipment.  Baseline  participants  were  not  concerned  with  equipment 
location. 

When  using  WhereNet  and  SSLC2  the  RFID/RTLS  technology  and  software  allowed 
participants  to  see  these  resources.  Most  participants  commented  they  did  not  want  to  be 
responsible  for  finding  a  piece  of  equipment  as  demonstrated  in  the  WhereNet  and 
SSLC2  conditions.  They  believed  this  specific  job  or  task  was  assigned  to  a  different 
individual  or  group  on  the  maintenance  team,  usually  AGE.  They  suggested  AGE  would 
be  interested  in  RFID/RTLS  location  information  for  equipment.  Some  participants 
indicated  they  liked  knowing  where  the  resources  were,  but  searching  would  take  them 
away  from  more  important  tasks.  It  is  important  not  to  add  additional  tasks  for  the 
Expeditors.  It  is  much  faster  for  them  to  call  for  the  equipment  and  they  will  not  use  the 
system  if  it  is  not  just  as  easy  as  their  current  techniques.  They  did  not  use  equipment 
location  to  provide  them  with  situation  awareness  information.  For  future  evaluations 
when  multiple  problems  are  presented  simultaneously  the  impact  of  moving  resources 
may  be  more  important  and  equipment  location  and  status  information  might  be  used  by 
Expeditors  as  a  situation  assessment  tool.  Participants  said  status  and  availability 
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information  specifically  for  low  density  high  demand  equipment  would  be  more 
beneficial  than  equipment  location  in  determining  equipment  allocation. 

WhereNet  and  SSLC2  provided  an  update  that  showed  equipment  moving.  While  this 
has  the  potential  to  provide  situation  awareness  information,  this  was  not  something  that 
participants  used  during  the  scenarios.  The  Expeditors  are  typically  moving  on  to  a 
different  task  and  assume  the  equipment  will  arrive  once  requested. 

5.6.1. 2  Personnel  Resources 

For  personnel  resources  there  are  high  frequency  counts  under  Baseline  and 
WhereNet  for  personnel  allocation.  Personnel  availability  is  also  higher  for  Baseline 
compared  to  WhereNet  and  SSLC2  (Table  11).  Location  was  similar  across  the  three 
conditions  with  higher  frequencies  under  WhereNet  and  SSLC2  because  participants 
could  search  for  people  using  the  software.  In  the  Baseline  condition  they  called  others  to 
find  people.  Many  participants  commented  that  finding  personnel  was  one  of  their 
biggest  problems.  They  stated  a  great  deal  of  time  was  wasted  looking  for  people  and 
believed  the  ability  to  find  personnel  would  be  very  beneficial.  They  expressed  concerns 
about  how  personnel  would  respond  to  the  concept  of  tagging  and  thought  that  for 
civilians  at  Continental  United  Status  (CONUS)  base  settings  it  may  be  a  problem; 
however  it  was  not  an  issue  for  deployment. 

Table  11:  Personnel  Information  Analysis 


Information 

Baseline 

WhereNet 

SSCL2 

Total 

Personnel  Allocation 

10 

15 

2 

27 

Personnel  Availability 

17 

5 

6 

28 

Personnel  Location 

8 

18 

12 

38 

Personnel  Qualifications 

1 

0 

0 

1 

Personnel  qualifications  was  another  area  in  which  participants  thought  information 
was  needed,  but  they  did  not  refer  to  it  in  the  scenarios.  One  reason  may  be  that  in  most 
cases  the  scenarios  did  not  require  any  very  specific  qualifications  beyond  crew  chief. 
Also,  as  soon  as  they  realized  it  was  not  available  from  any  source  in  the  study  they  did 
not  request  it.  While  they  did  not  refer  to  this  often  in  the  scenarios  they  did  comment 
the  information  was  important.  In  the  post  questionnaire  when  asked  to  rate  the 
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importance  of  information,  eleven  of  fourteen  rated  personnel  skill  level  as  a  7  (extremely 
important). 

5.6. 1.3  Other  Information 

Another  high  frequency  item  was  Aircraft  Problem.  This  category  refers  to 
information  about  the  other  Non  Mission  Capable  (NMC)  aircraft  on  the  flightline,  not 
the  aircraft  they  were  currently  working  in  the  scenario.  Participants  asked  for  more 
information  related  to  the  problems  to  determine  if  any  of  those  aircraft  would  be 
available  for  a  swap.  This  request  was  made  more  frequently  under  WhereNet  and 
SSLC2  conditions  than  Baseline.  The  reason  for  this  difference  is  not  clear.  However,  it 
does  relate  to  situational  awareness.  Participants  were  attempting  to  find  more 
information  about  the  overall  flightline  to  help  them  make  their  decision.  This  was  the 
case  even  under  the  SSLC2  condition  where  a  recommendation  for  which  aircraft  to  swap 
with  was  provided. 

Other  high  frequency  information  items  were  aircraft  status  (which  includes  any 
aircraft  on  the  flightline),  current  time,  ETIC,  take-off  time,  and  task  assignment.  Task 
assignment  refers  to  when  they  assigned  a  specific  task  (e.g.  calling  for  a  weapons 
download,  or  defuel).  Swap  was  also  high  frequency  because  they  were  asked  to  make 
the  fix/swap  decision  and  in  most  cases  the  scenario  required  a  swap. 

5.6.2  Source 

Where  the  participants  sought  information  was  documented  and  the  frequency  was 
tabulated.  Section  3.2.2  of  Appendix  B,  Data  Collection  illustrates  this  data.  In  the 
Baseline  and  WhereNet  conditions  information  was  primarily  sought  from  AGE,  AMU 
Dispatch,  the  Specialist,  and  status/schedule  sheet.  If  participants  were  confused  they 
would  sometimes  request  information  or  clarification  about  the  scenario  via  the  Principal 
Investigator.  The  category  labeled  “self’  indicates  they  used  experience  and  often 
computed  the  time  to  site  and  time  to  fix  mentally  to  come  up  with  those  data  points.  The 
data  were  not  readily  available  without  this  mental  calculation.  Mental  calculation  was 
not  necessary  in  SSLC2,  resulting  in  low  frequency  count  under  that  category. 

Under  the  WhereNet  condition,  participants  referred  to  the  WhereNet  screens  41 
times,  yet  they  also  sought  information  from  the  same  resources  as  Baseline,  often  in  the 
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same  frequency.  This  is  not  surprising  because  WhereNet  only  provides  resource 
location.  All  other  information  needed  to  make  decisions  must  be  found  from  other 
resources.  Thus  WhereNet  is  unlikely  to  reduce  workload  or  to  enhance  decision  support. 

In  the  case  of  SSLC2,  all  information  was  provided  on  the  system  so  participants 
referred  to  fewer  additional  sources.  However,  the  simulation  scenario  required  the 
resource  problems  to  be  called  in  via  radio  by  the  SME  acting  as  a  specialist  after  the  first 
initial  fix/swap  decision.  Sometimes  the  participant  forgot  what  the  specialist  requested 
and  called  back.  The  information  source  for  SSLC2  was  specified  by  screen.  They  sought 
their  information  from  the  overview  screens,  the  problem  schedule  screen,  and  the 
comparison  screens.  They  also  used  select  plan  in  order  to  make  the  fix/swap  decision. 
SSLC2  significantly  reduced  the  radio  communication  traffic  as  participants  were  able  to 
look  for  information  on  the  system  rather  than  via  radio.  SSLC2  provided  the  Expeditor  a 
tool  that  centralizes  information  and  has  potential  for  increasing  situation  awareness.  If  a 
great  deal  of  the  information  must  be  sought  via  radio,  paper  sheets,  visual  input,  etc, 
participants  must  integrate  all  the  information  in  their  mind.  SSLC2  reduces  the  need  for 
mental  integration  of  information. 

5.6.3  Destination 

Frequencies  were  determined  for  Destination,  which  refers  to  where  the  participant 
sent  or  distributed  the  information  and  can  be  found  in  Section  3.2.3  of  Appendix  B,  Data 
Collection.  In  the  Baseline  and  WhereNet  conditions,  participants  sent  information  to  10 
sources,  with  a  count  of  65  and  70  times  information  was  sent.  The  primary  destinations 
were  AGE,  AMU  dispatch,  MOC,  and  the  Specialist.  For  the  SSLC2  condition,  only 
AGE,  AMU  dispatch  and  MOC  were  destination  sources  for  radio  calls,  and  the 
frequency  count  was  4.  Again,  the  WhereNet  and  Baseline  conditions  required 
participants  to  send  tasks  and  information  to  various  destinations  via  the  radio,  whereas 
SSLC2  did  this  automatically. 

5.6.4  Cognitive  Task  Analysis 

An  objective  of  the  Scientific  Study  was  to  validate  the  cognitive  models  presented  in 
Section  3.  This  study  focused  specifically  on  one  Expeditor  task,  “Assess  aircraft  status”, 
focusing  on  the  fix  or  swap  decision  process.  Information  from  the  verbal  protocol  under 


61 


the  Process  category  was  analyzed  and  results  were  compared  to  the  cognitive  model 
created  from  the  initial  data  collection  via  interviews. 

A  table  was  created  indicating  the  general  steps  participants  used  during  the  Scientific 
Study  for  both  Fighter  and  Airlift.  Once  the  table  was  developed,  the  steps  and  the  order 
of  steps  each  participant  described  in  their  verbal  protocol  were  placed  in  the  table.  The 
frequencies  for  the  various  task  steps  were  then  calculated.  The  number  of  times  a  step 
was  the  first  step,  second  step,  third  step,  etc.  was  also  counted.  Tables  12  and  13 
provide  the  frequency  counts  for  the  various  steps  for  Fighter  and  Airlift  participants 
during  the  initial  scenario  trigger  requiring  participants  to  determine  the  problem  and 
decide  whether  to  fix  or  swap. 
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Table  12:  Airlift  Frequencies  for  Each  Condition 


Frequency 

SS 

- 

CO 

CO 

x — 

CD 

x— 

- 

- 

CO 

WN 

T— " 

CD 

CD 

CM 

CO 

CM 

VN 

- 

CO 

x — 

CO 

- 

BL 

- 

CO 

CM 

CD 

CD 

CO 

CD 

NA 

co 

CM 

- 

ID 

x — 

Task  List 

Compare  time  to  fix  to  flight  schedule 

1)  Current  time 

2)  Take  off  time 

3)  Will  it  make  flight? 

Compare  time  to  swap 

Decide  to  swap 

•  Determine  aircraft  to  swap 

•  Update  ProSuper 

•  Update  MOC 

•  Get  Job  Number 

•  Select  Swap  Recommendation 

Begin  Fix 

•  Defuel 

•  Flares  (Airlift  only) 

•  Jacks 

•  Tow  Team 

•  Review  time  to  site 

•  Weapons  (fighter  only) 

•  Egress  (fighter  only) 

Review  how  swap  affects  overall 
schedule 

Frequency 

SS 

x — 

T— 

x — 

- 

x — 

x— 

CO 

CM 

CD 

WN 

- 

- 

CM 

co 

CM 

- 

CD 

co 

NA 

BL 

CM 

CM 

CM 

- 

x — 

- 

x — 

- 

CD 

VN 

Task  List 

Begin  analysis  of  situation 

•  Hazard/safety 

•  Flares/weapons 

•  Hanger 

•  Parts 

•  Weather 

•  Fuels 

o  Download  necessary? 

o  availability 

•  Fuel  barn  location 

•  Jack  (time,  outside?) 

•  Aircraft  configuration 

•  Cargo/mission 

•  Other  broken  a/c  status 
update 

Determine  Time  to  Fix 

•  From  specialist  or  crew 
chief 

•  Self  estimate 

•  SSLC2 
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Table  13:  Fighter  Frequencies  by  Condition 


Frequency 

SS 

to 

1 

CM 

B 

1 

1 

1 

CO 

WN 

X — 

1 

1 

1 

1 

VN 

— 

x — 

CM 

CO 

CO 

BL 

- 

CO 

1 

1 

1 

NA 

x — 

- 

CM 

- 

Task  List 

Compare  time  to  fix  to  flight  schedule 

4)  Current  time 

5)  Take  off  time 

6)  Will  it  make  flight? 

Compare  time  to  swap 

Decide  to  swap 

H 

•  Update  ProSuper 

•  Update  MOC 

H 

•  Select  swap  recommendation 

Begin  Fix 

•  Defuel 

•  Flares  (Airlift  only) 

•  Jacks 

•  Tow  Team 

•  Review  time  to  site 

•  Weapons  (fighter  only) 

•  Egress  (fighter  only) 

Frequency 

SS 

1 

i 

1 

1 

CO 

WN 

x — 

fl 

1 

x — 

CM 

CO 

VN 

BL 

1 

1 

CM 

CO 

CM 

NA 

Task  List 

Begin  analysis  of  situation 

•  Hazard/safety 

•  Flares/weapons 

•  Parts 

•  Weather 

•  Fuels 

o  Download 
necessary? 

o  availability 

•  Fuel  barn  location 

•  Jack  (time,  outside?) 

•  Aircraft  configuration 

•  Cargo/mission 

•  Other  broken  a/c  status 
update 

Determine  Time  to  Fix 

•  From  specialist  or  crew 
chief 

•  Self  estimate 

•  SSLC2 
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To  visualize  the  process  steps,  including  order,  flow  diagrams  were  created  for  each 
condition  under  each  aircraft  group  type  (Fighter  and  Airlift).  A  major  task  is  illustrated  as  a 
square  with  the  subtasks  making  up  each  task  indicated  as  circles  within  the  square.  The  legend 
for  each  subtask  is  at  the  bottom  of  the  figure.  Each  colored  line  represents  a  different  subject. 
The  starting  location  is  indicated  by  the  thickest  line  for  each  participant.  It  must  be  noted  while 
each  step  is  delineated  in  the  flow,  participants  often  made  decisions  simultaneously.  For 
example,  to  specify  they  had  decided  to  swap,  they  might  say  “I  will  swap  with  tail  451”.  This 
was  indicated  in  the  diagram  as  a  line  going  to  the  Swap  decision  box  first,  then  the  subtask  of 
“determine  a/c  to  swap”.  A  discussion  of  the  process  patterns  by  aircraft  type  and  the  flow 
diagrams  arc  presented  below.  Figures  16-18  depict  Fighter  participants’  process  flows  by 
technology  condition  while  Figures  19-21  depict  Airlift  participants’  process  flows  by 
technology  conditions. 

5.6.4.1  Fighter  Participants 

Fighter  participants  using  SSLC2  did  not  evaluate  the  specific  aircraft  problem  in  depth.  As 
indicated  by  Figure  16  and  the  frequency  counts,  there  are  no  references  in  their  protocols  related 
to  pre-decision  analysis  when  using  SSLC2.  In  general,  the  process  was  to  look  at  the  time  to 
fix,  determine  take-off  time  and  immediately  make  the  fix/swap  decision,  and  use  “select  plan” 
to  execute  the  swap.  One  participant  determined  which  aircraft  to  swap  with  before  going  with 
the  recommended  plan.  Two  subjects  also  looked  at  swap  time.  No  participants  began  the  fix 
before  making  the  swap  decision.  Four  of  the  six  subjects  made  the  swap  decision  within  three 
steps,  and  two  within  four  steps. 

Recall  that  SSLC2  gave  the  participant  the  current  schedule,  time  to  fix,  time  to  swap  and  a 
recommendation  for  which  aircraft  to  swap  with.  They  were  not  required  to  determine  individual 
steps  toward  the  fix  or  mentally  calculate  tasks  times  to  determine  a  total  fix  time.  They  did  not 
need  to  use  the  radio  to  send  out  messages  to  begin  the  fix,  resulting  in  no  frequency  counts  in 
the  Begin  Fix  task.  Their  level  of  communication  using  the  radio  was  reduced  compared  to 
Baseline  and  WhereNet  conditions.  Participants  seemed  to  trust  what  SSLC2  recommended. 
Only  one  participant  in  this  group  went  back  to  consider  how  the  swap  might  affect  the  overall 
schedule. 
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Process  flow  patterns  for  WhereNet  and  Baseline  conditions  are  different  from  SSLC2. 
Under  both  these  conditions  the  aircraft  status  and  flight  schedules  were  available  on  their  paper 
status  sheets.  WhereNet  provides  resource  location  but  no  other  information  related  to  the  fix  or 
swap.  Because  these  two  conditions  are  similar  to  how  they  currently  work  on  the  flightline  we 
would  expect  the  WhereNet  and  Baseline  conditions  to  be  very  similar.  Yet  it  appears  that 
participants  behaved  somewhat  differently. 

For  the  WhereNet  condition  three  of  the  six  subjects  began  by  determining  the  time  to  fix 
based  on  information  from  the  specialist.  One  participant  immediately  made  the  decision  to 
swap  based  on  his  experience.  One  participant  began  an  analysis  of  the  situation.  One  participant 
started  by  requesting  a  tow  team  before  requesting  time  to  fix.  Five  of  the  six  participants  made 
the  swap  decision  within  the  first  three  steps.  All  participants  had  to  determine  which  aircraft  to 
swap  with.  One  participant  reviewed  the  situation  after  the  decision.  Two  of  the  six  began  at  least 
one  step  in  the  fix  process  before  making  the  swap  decision.  Participants  did  not  discuss  the  time 
to  swap. 

In  the  Baseline  condition  the  processes  seemed  to  vary  more  by  participant.  For  the  Baseline 
condition  two  of  the  participants  made  the  fix/swap  decision  within  three  steps,  one  in  four  steps 
and  two  within  five  steps.  Three  of  the  subjects  began  with  some  analysis  of  the  situation,  and 
four  subjects  began  the  fix  process.  Three  of  the  six  began  the  fix  process  before  making  the 
swap  decision. 

It  appears  the  slight  difference  between  WhereNet  and  Baseline  is  that  more  participants  in 
Baseline  performed  a  pre-analysis  and  began  fix  tasks.  More  Baseline  participants  referred  to  the 
take-off  time  as  well.  These  differences  may  be  due  to  the  fact  participants  were  more 
comfortable  in  the  Baseline  condition.  When  using  WhereNet  they  may  have  spent  their  time 
locating  resources  without  gaining  much  information  necessary  for  the  specific  decision.  Again, 
participants  did  not  discuss  the  time  to  swap. 
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Figure  16:  Fighter  Participants'  Process  Flow  Using  SSLC2 


Begin  Analysis  of  Situation 
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Figure  17:  Fighter  Participants'  Process  Flow  Using  WhereNet 


Determine  Time  to  Fix 
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Figure  18:  Fighter  Participants'  Process  Flow  Using  Baseline 


5.6.4.2  Airlift  Participants 

As  with  Fighter  participants  the  general  process  for  Airlift  participants  while  using  SSLC2 
was  to  locate  the  time  to  fix,  compare  time  to  swap  and  then  make  the  fix/swap  decision  using 
“select  plan”.  Six  of  the  nine  participants  made  the  fix/swap  decision  within  three  steps.  Five 
participants  considered  at  least  one  item  related  to  pre-analysis  of  the  situation  before  making  the 
fix/swap  decision,  although  only  one  participant  analyzed  more  than  one  item  (Figure  19).  Only 
one  of  the  nine  participants  began  a  portion  of  the  fix  task  before  making  the  fix/swap  decision. 
Three  participants  reviewed  the  situation  after  making  the  decision.  Three  participants  compared 
time  to  swap.  Although  SSLC2  provides  a  swap  recommendation  at  least  four  of  the  participants 
specified  or  mentioned  which  aircraft  to  swap.  Two  of  those  four  considered  it  before  choosing 
“select  plan”  and  two  after  choosing  “select  plan”. 

In  the  WhereNet  and  Baseline  conditions  (Figures  20  and  21),  participants  spent  more  time 
doing  pre-analysis  of  the  situation  than  when  they  used  SSLC2.  They  also  spent  time  beginning 
the  fix.  For  the  WhereNet  condition,  five  of  the  nine  participants  made  the  fix/swap  decision 
within  three  steps.  Six  of  the  nine  participants  did  some  pre-analysis  before  making  the  decision. 
Two  of  the  nine  began  the  fix  before  making  the  swap  decision.  Participants  did  not  discuss  time 
to  swap. 

The  Baseline  condition  was  similar  to  WhereNet  with  four  of  the  nine  making  the  fix/swap 
decision  within  three  steps.  Five  of  the  nine  did  some  pre-analysis  before  making  the  decision 
and  three  began  the  fix  before  making  the  swap  decision.  As  with  WhereNet,  participants  did  not 
discuss  time  to  swap. 
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Figure  19:  Airlift  Participants'  Process  Flow  Using  SSLC2 


Begin  Analysis  of  Situation 
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Figure  20:  Airlift  Participants'  Process  Flow  Using  WhereNet 


Begin  Analysis  of  Situation 
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Figure  21:  Airlift  Participants'  Process  Flow  Using  Baseline 


5.6.5  Validation  of  the  Cognitive  Model 

One  purpose  of  the  Scientific  Study  was  to  validate  the  cognitive  task  analysis  presented  in 
Section  3.2.  The  Scientific  Study  focused  on  the  task  of  assessing  aircraft  status  and  the  fix/swap 
decision.  The  original  cognitive  task  analysis  OFM  diagram  for  Assess  Aircraft  Status  shown  in 
Section  2.3.2  of  Appendix  B  provides  the  OFM-COG  analysis.  The  results  of  the  verbal  protocol 
indicate  the  general  process  described  by  the  OFM-COG  is  accurate.  The  Scientific  Study 
breaks  down  the  fix/swap  decision  in  more  detail.  Figures  22  is  a  new  OFM  diagram  detailing 
the  verbal  protocol  results  based  on  the  WhereNet  and  Baseline  conditions  (combined)  from  the 
point  in  the  process  where  the  problem  has  been  accurately  assessed.  Table  14  is  the  new  OFM- 
COG  analysis.  The  OFM  diagram  and  OFM-COG  analysis  based  on  verbal  protocols  when  using 
SSLC2  are  presented  in  Figure  23  and  Table  15. 


75 


1.  Analyze  information 

2.  Start  fix  or  make  final  fix/swap  decision 

3.  Fix  started 

4.  Swap  chosen 

5.  Notify  personnel 

6.  Execute  decision 

7.  Resources  required 

8.  Resources  identified 

9.  Resources  available  via  other  units  and/ 
or  owners. 

10.  Resources  need  pickup 

1 1 .  Prepare  a/c  for  next  sortie 


2 

OR 


Figure  22:  Fix/Swap  Decision  OFM  Diagram  for  WhereNet  and  Baseline 
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1 .  Problem  accurately  assessed 

2.  Information  analyzed 

3.  Fix  or  Swap  chosen 

4.  Fix  and  swap  plans  specified,  appropriate 
personnel  notified, 

5.  Fix  and  swap  tasks  proceeding 

6.  Change  of  selected  resources  needed. 

7.  Resources  identified 

8.  Resources  reassigned 


Figure  23:  Fix/Swap  Decision  OFM  diagram  for  SSLC2 
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Table  15:  "Assess  Aircraft  Status"  Function  Updated  OFM-COG  Analysis  for  SSLC2 
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Because  SSLC2  provides  fix  and  swap  task  information,  assigns  the  personnel  and 
equipment  and  automatically  notifies  personnel  to  begin  the  fix  and  swap  tasks,  many 
Expeditor  subtasks  (determine  aircraft  to  swap,  notifying  personnel,  determining 
resources)  are  eliminated  under  the  SSLC2  condition.  During  the  fix/swap  decision 
process  participants  did  not  spend  time  trying  to  reassign  resources  to  the  problem  once 
SSLC2  made  selections.  Several  participants  indicated  they  were  likely  to  make  such 
changes  based  on  their  knowledge  of  personnel  and  their  skills. 

Incorporating  a  computer  based  technology  system  like  SSLC2  into  the  maintenance 
process  changes  how  the  tasks  are  carried  out  and  the  type  of  cognitive  processing 
required  by  the  Expeditor.  Time  is  spent  detecting  and  reviewing  information  presented 
by  the  system  rather  than  searching  for  information  from  a  wide  variety  of  sources  and 
mentally  integrating  the  task  steps  and  times. 

A  very  in-depth  task  analysis  specifying  each  screen  and  computer  input  when  using 
SSLC2  was  not  conducted  because  the  system  design  was  not  being  tested,  nor  is  this 
specific  design  being  recommended.  While  SSLC2  was  able  to  provide  additional 
information,  the  interface  required  too  many  steps  to  carryout  the  tasks.  However,  the 
results  show  there  can  be  reduced  need  for  interruptive  radio  communication  and  mental 
integration  and  many  participants  were  open  to  the  use  of  such  a  technology. 

5.7  SUMMARY  OF  RESULTS  AND  DISCUSSION 

The  Expeditors  primary  goal  is  to  facilitate  maintenance  so  sorties  can  be  met.  The 
study  focused  on  the  task  of  assessing  aircraft  status  and  the  fix/swap  decision.  As  stated 
in  the  Section  4.0  the  objectives  of  the  Scientific  Study  were  to: 

1)  Validate  the  cognitive  model  developed  from  data  collection  interviews  through 
verbal  protocol  (source)  techniques  and  to  determine  what  information  is  most 
useful  to  help  Expeditors  make  flightline  decisions. 

2)  Evaluate  user  performance  and  opinions  on  enhanced  data  streams. 

5.7.1  Summary  of  Objective  1 :  Validating  the  Cognitive  Model 

The  results  of  the  study  indicate  that  the  cognitive  representations  presented  in 

Section  3  were  accurate.  The  OFM  diagrams  in  Section  2.3.2  of  Appendix  B  show  that 
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the  Expeditor  reviews  a  variety  of  information  input  which  leads  to  problem 
identification.  Once  the  problem  is  identified,  the  Expeditor  has  the  maintenance 
problem  assessed  by  a  specialist.  The  study  took  place  from  the  point  where  the 
assessment  had  been  made. 

Upon  assessment  the  next  task  is  to  determine  how  the  sortie  can  be  met,  by  fixing, 
swapping  or  using  a  spare  aircraft.  Using  a  spare  aircraft  is  not  a  swap  if  it  is  on  the  flying 
schedule.  If  it  is  not  on  the  schedule  it  is  considered  a  swap  and  a  deviation  to  the  flying 
schedule.  Resources  needed  are  determined  and  located  and  the  Expeditor  continues  to 
monitor  maintenance  progress  to  continue  toward  the  goal  of  meeting  the  flying  schedule. 
The  results  show  that  this  is  the  general  process  participants  took  during  all  three 
conditions  so  the  basic  model  is  accurate.  Additionally,  more  specific  information  was 
provided  showing  that  strategies  differed  among  participants  and  between  aircraft  type 
(airlift  vs  fighter).  Once  the  problem  is  identified  and  assessed  by  a  Specialist,  the 
Expeditor  quickly  makes  the  fix/swap  decision  focusing  primarily  on  the  time  to  fix  and 
the  flight  schedule.  The  updated  OFM  diagram  (Figure  23)  breaks  down  the  process  in 
more  detail  as  does  the  OFM-COG  table  (Table  14).  Flowever,  different  participants  had 
different  strategies,  with  some  participants  reviewing  the  situation  more  in-depth  and 
some  beginning  tasks  for  the  fix  before  making  a  fix/swap  decision.  The  most  interesting 
finding  is  that  when  the  SSLC2  decision  support  tool  is  introduced,  the  participants  do 
change  their  process.  In  this  case,  participants  no  longer  needed  to  determine  the  specific 
tasks  and  how  long  it  would  take  overall  to  fix  the  aircraft,  reducing  the  need  for  mental 
calculations  and  the  need  to  call  a  variety  of  sources  for  information.  Participants 
followed  the  SSLC2  recommendation  to  fix  or  swap  and  which  aircraft  to  swap  with. 
Use  of  this  system  in  a  deployed  setting  when  they  might  be  more  familiar  with  resources 
may  result  in  more  time  spent  making  changes  to  the  fix  tasks,  times,  and  resources 
assigned.  Or  they  may  just  trust  the  recommendation.  During  this  simulation  they  were 
not  familiar  with  all  available  resources  and  personnel  so  they  took  the  information  at 
face  value. 

Another  validation  of  the  model  is  that  when  using  the  SSLC2  simulation  participants 
commented  on  pieces  of  information  that  were  not  available  in  the  SSLC2  simulation 
system,  personnel  skill  level  and  certifications.  They  did  not  specifically  search  for  the 
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information,  but  they  stated  that  it  would  be  beneficial,  especially  if  they  did  not  know 
everyone  on  the  flightline.  The  fact  that  they  did  not  request  additional  information 
indicated  the  initial  simulation  system  was  able  to  capture  important  information  needed 
for  the  decision  process. 

5.7.2  Summary  of  Objective  2:  User  Opinions  On  Enhanced  Data  Streams 

User  opinions  on  the  enhanced  data  streams  were  elicited  through  questionnaires 

at  the  end  of  each  condition  and  at  the  end  of  the  study.  Both  the  WhereNet  and  SSLC2 
condition  have  enhanced  data  streams  showing  resource  location.  SSLC2  has  additional 
data  including  aircraft  status  and  location,  aircraft  maintenance  schedules,  daily  flying 
schedules,  personnel  and  equipment  resources  and  their  locations,  tasks  necessary  to  fix 
the  broken  aircraft,  tasks  necessary  to  swap  an  aircraft,  time  to  complete  fix  tasks,  time  to 
get  resources  to  the  fix  site,  resource  types  necessary  to  complete  tasks,  specific  resources 
assigned  to  tasks,  and  a  recommendation  to  fix  or  swap  an  aircraft. 

The  ranking  data  indicated  that  participants  preferred  SSLC2  to  both  WhereNet 
and  Baseline.  However,  they  preferred  Baseline  to  WhereNet.  These  results  show  that 
participants  prefer  advanced  data  streams  if  the  information  is  presented  in  a  way  that 
helps  with  their  tasks.  While  WhereNet  provided  enhanced  data,  many  participants  felt 
that  WhereNet  did  not  do  a  good  job  of  presenting  the  information  in  a  usable  way, 
resulting  in  information  overload. 

When  participants  rated  15  statements  related  to  the  use  of  the  technology,  their 
overall  ratings  for  SSLC2  were  higher  than  both  WhereNet  and  Baseline.  WhereNet 
ratings  were  higher  than  Baseline.  The  average  rating  for  Baseline  is  still  high  (X  =  5.0) 
indicating  on  average  participants  slightly  agreed  with  the  statements  overall. 

Several  statements  on  the  post  condition  questionnaire  and  post  test  ranking 
questionnaire  were  related  to  monitoring  the  status  of  resources,  locating  both  equipment 
and  personnel  resources,  and  availability  of  both  equipment  and  personnel.  In  general, 
the  results  showed  users  preferred  SSLC2  to  both  WhereNet  and  Baseline.  They  rated 
both  SSLC2  and  WhereNet  better  than  the  Baseline  for  these  resource  activities. 
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5.8  EXPERIMENTER  OBSERVATIONS 

During  the  study  experimenters  made  notes  about  their  observations.  These 
observations  along  with  suggestions  and  feedback  provided  by  the  participants  are 
summarized  below. 

5.8.1  SSLC2  Functionality 

Participants  made  positive  comments  about  the  ability  to  get  an  overview  of  the 
flightline,  both  geographically  and  as  a  schedule,  and  liked  being  able  to  dig  deeper  into 
aircraft  problems  and  tasks  to  be  worked.  Participants  frequently  worked  between  the  top 
level  and  lower  level  views,  specifically  to  see  how  the  aircraft  problems  affected  the 
overall  flightline  schedule.  Participants  varied  in  how  frequently  they  moved  between  the 
views  depending  on  their  current  maintenance  role.  Those  concerned  with  solving  aircraft 
problems  tended  to  work  within  the  problem  schedule  or  geographic  views  while  others, 
such  as  ProSupers,  looked  for  potential  schedule  impacts. 

The  Scientific  Study  focused  on  one  aircraft  problem  needing  attention;  however  each 
scenario  had  another  aircraft  that  was  NMC  as  well.  The  specific  task  information  to 
work  that  problem  was  not  provided  in  the  scenario.  Expeditors  and  ProSupers  expect  to 
multitask  and  need  to  respond  to  more  than  one  problem  at  a  time.  Frequently  the 
participants  wanted  to  work  all  aircraft  problems  to  see  how  they  affected  the  flying 
schedule  and  spare  availability.  Participants  suggested  that  a  future  system  give  them  the 
ability  to  multitask  and  give  information  about  every  aircraft  to  help  with  decision 
making.  Multiple  problems  will  be  investigated  in  the  next  Spiral. 

SSLC2  also  broke  each  problem  into  a  series  of  tasks  to  be  completed.  Each  task 
listed  associated  equipment  and  personnel  resources  with  their  time  to  site  and  time  to 
fix.  Participants  liked  seeing  the  list  of  tasks  and  said  they  would  use  this  feature  on  the 
real  flightline  where  they  know  their  resources.  Participants  said  they  would  like  the 
ability  to  create  or  modify  the  tasks  needed  to  fix  problems.  The  entire  listing  of  tasks 
and  resources  was  overwhelming  to  some  participants  so  they  suggested  minimizing 
tasks  once  they  are  completed  to  avoid  scrolling  through  tasks  already  accomplished. 
Participants  sometimes  referred  to  the  wrong  task  when  trying  to  change  a  resource. 
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The  scenarios  identified  flightline  equipment  and  personnel  resource  problems 
needing  attention.  Participants  were  asked  to  identify  a  new  piece  of  equipment  or 
person  to  replace  the  problem  resource.  Within  the  SSLC2  condition  participants  needed 
to  work  through  a  series  of  screens  to  find  and  replace  a  resource.  Adding  comparisons 
generally  required  seven  to  eight  steps.  Many  participants  commented  that  the  capability 
of  changing  resources  was  necessary  however  the  current  functionality  has  too  many 
steps.  They  said  the  information  provided  such  as  the  time  to  site,  resource  location  and 
names  were  helpful.  Suggestions  included  reallocating  new  resources  from  the  problem 
geographic  view  or  original  task  lists.  Some  participants  suggested  a  click  and  point 
interface  rather  than  dialog  boxes  and  pull  downs. 

Additional  items  and  comments  highlighting  positive  aspects  of  SSLC2  functionality 
included  the  ability  to  zoom  in  and  out  within  the  geographical  views.  Some  participants 
used  this  feature  to  view  all  resources  on  the  entire  flightline  while  others  zoomed  into 
the  problem  aircraft  and  resources  associated  with  the  fix.  This  functionality  allowed 
participants  to  tailor  views  based  on  their  individual  needs. 

Suggestions  for  functionality  enhancements  included  incorporating  the  positive 
aspects  of  WhercNct,  particularly  the  ability  to  filter  by  resource  types  or  finding  specific 
resources  through  a  drop  down  menu  or  wildcard  selection.  The  filter  ability  would  allow 
users  to  see  only  the  details  necessary  and  eliminate  the  problem  of  too  much 
information. 

Participants  had  numerous  concerns  with  hardware  functionality.  Many  participants 
said  the  current  version  of  SSLC2  was  optimal  for  someone  at  a  desk  and  not  in  a  truck, 
the  typical  place  of  an  Expeditor.  Participants  said  any  system  would  have  to  be  portable, 
ruggedized  and  wireless. 

5.8.2  Validation  and  Alerts 

Expeditors  currently  use  two-way  communication  using  radios.  When  they  make 
a  request  they  are  given  a  confirmation  that  the  information  has  been  received,  and 
continue  to  receive  information  updates.  The  simulation  did  not  include  the  two-way 
communication.  Many  participants  stated  that  they  would  expect  validation  from  the 
flightline  resources  that  messages  were  received  from  SSLC2.  The  only  feedback  in  the 
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simulation  was  movement  of  resources,  however,  they  did  not  watch  for  the  movement  as 
they  are  busy  moving  to  other  tasks.  Possible  validation  considerations  include 
communicating  flying  schedule  and  resource  changes  to  those  on  the  flightline, 
monitoring  when  tasks  are  complete  and  sending  resources  to  site. 

Participants  were  also  concerned  maintainers  would  not  monitor  the  computer 
screen  for  all  changes  so  some  other  alert  or  validation  is  necessary  to  know  they 
received  the  changes. 

5.8.3  Tagging  Personnel 

Participants  thought  the  ability  to  see  personnel  location  could  be  extremely 
important  on  the  flightline.  Many  participants  indicated  that  the  biggest  difficulty  they 
have,  particularly  as  ProSupers  and  Expeditors,  is  finding  people.  Many  indicated  that  a 
large  amount  of  time  is  wasted  searching  for  personnel.  They  indicated  that  tagging 
identifications  could  be  especially  useful  when  you  needed  someone  with  specific 
certifications  quickly.  One  challenge  to  tagging  personnel  resources  is  that  the  Expeditor 
and  ProSuper  are  not  always  the  direct  supervisor  of  all  flightline  personnel.  They  do  not 
necessarily  know  what  task  a  Specialist  has  been  assigned.  Other  challenges  are  social 
including  the  concern  with  privacy  as  well  as  issues  of  trust.  Many  bases,  particularly 
Guard  and  Reserve  Units,  have  unions  and  the  civilian  culture  would  be  more  resistant  to 
RFID  tagging.  However,  during  deployed  missions  this  is  not  an  issue  and  may  be 
extremely  beneficial  during  wartime  operations. 

5.8.4  Sensors  and  Additional  RFID  Considerations: 

The  SSLC2  system  provided  a  simulation  of  RFID/RTLS  technology  to  track 

personnel  and  equipment  location.  Participants  were  asked  what  additional  information 
should  be  collected  through  sensors  that  would  impact  flightline  decisions.  They 
indicated  that  information  such  as  base  supplies  on  the  Quick  Reference  List,  equipment 
status  (FMC/NMC),  fuel  level,  oil  level,  liquid  oxygen  level,  and  usage  status  (in  use  or 
not  in  use)  would  be  useful.  All  of  these  items  were  mentioned  numerous  times  at  all 
field  test  locations.  Participants  said  this  information  could  aid  in  making  cannibalization 
and  fix/swap  decisions.  RFID/RTLS  and  sensor  information  could  help  participants 
prioritize  tasks  and  resources. 


87 


Tagging  every  resource  on  a  flightline  could  be  overwhelming.  Participants 
verbalized  that  WhereNet  had  too  much  information  at  times  because  everything  was 
tagged  and  the  most  critical  information  was  not  readily  accessible.  Participants  at  Ft. 
Wayne  suggested  tagging  Low  Density/High  Demand  (LD/HD)  items,  such  as  mules  and 
jacks.  Participants  indicated  there  are  resources  either  in  supply  or  frequently  used  or 
borrowed  that  would  be  more  important  to  tag. 

6  Recommendations 

Spiral  One  of  the  SSLC2  program  was  successful  in  providing  input  toward 
requirements  specifications.  The  Scientific  Study  helped  to  determine  concepts  for 
SSLC2,  showed  that  enhanced  data  streams  can  help  support  flightline  decisions,  and 
provided  information  on  important  data  elements  to  include  in  a  system.  This  section  first 
outlines  some  recommendations  based  on  the  output  of  all  Spiral  One  phases.  The  initial 
requirements  are  then  briefly  presented.  Future  research  questions  related  to  the  concepts 
of  SSLC2  are  then  discussed  followed  by  plans  for  Spirals  Two  and  Three. 

6.1  SPECIFIC  RECOMMENDATIONS 

The  research  conducted  to  date  provides  input  for  recommendations  for  SSLC2 
concepts.  Recommendations  are  discussed  below. 

6.1.1  Sensor  Information 

Results  of  this  effort  have  shown  that  the  incorporation  of  real  time  sensing 
technologies  can  improve  flightline  logistics  support.  For  this  spiral  RFID/RTLS  was 
simulated  to  show  resource  (equipment  and  personnel)  location  information.  This 
information  along  with  other  decision  support  information  was  presented  to  Expeditors 
and  the  overall  result  was  positive.  The  results  also  pointed  out  additional  information 
used  in  decision  making  that  could  be  sensed  and  integrated  and  are  listed  below. 

•  Supplies  on  the  quick  reference  list; 

•  Equipment  Status  (broken,  in  use,  not  in  use); 

•  Equipment  fuel  level; 

•  Liquid  oxygen  level; 

•  Temperature; 
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•  Battery  level;  and 

•  Oil  levels. 

Sensing  supplies  on  the  quick  reference  list  (supplies  on  base  only)  was  an  important 
element  to  most  participants.  The  ability  to  quickly  determine  if  those  supplies  were 
available  would  shorten  the  decision  process.  Additional  sensed  information  should  be 
evaluated  to  determine  how  and  when  they  would  be  used  so  that  it  can  be  integrated  into 
the  system  effectively.  Efficient  use  of  sensors  and  integration  of  the  information  is 
critical  and  the  potential  for  information  overload  is  very  high.  It  is  recommended  that 
information  related  to  LD/HD  items  is  sensed. 

6.1.2  Portability 

As  mentioned  previously,  the  SSLC2  system  would  have  to  be  hand  held  or  highly 
portable.  Most  participants  indicated  the  laptop  system  would  not  be  feasible  for  an 
Expeditor.  It  is  highly  recommended  a  handheld  portable  device  be  used  for  SSLC2. 
The  concept  of  incorporating  both  the  visualized  information  and  verbal  communication 
in  one  unit  should  be  explored.  Perhaps  Voice  Over  Information  Protocol  (VOIP)  could 
be  incorporated.  It  is  highly  unlikely  that  SSLC2  would  eliminate  the  need  for  some 
verbal  communication  between  team  members  on  the  flightline.  The  concept  is  to  create 
one  system  the  Expeditor  believes  he  can  not  do  without. 

6.1.3  Direct  Manipulation  Interface 

SSLC2  should  have  a  direct  manipulation  interface  to  include  features  such  as  point 
and  click  interactive  visualizations  and  a  drag  and  drop  interface.  These  types  of  features 
minimize  the  information  displayed  to  just  the  pertinent  information  required,  and 
eliminated  unnecessary  information.  They  allow  the  user  to  tailor  the  interface  to  their 
preferences  and  decision  support  needs.  These  also  minimize  the  number  of  steps  and 
screens  required  to  reach  the  required  information. 

6.1.4  Tailored  to  Different  User  Types 

As  the  Scientific  Study  results  indicated,  participants  made  their  decisions  in  a 
number  of  different  ways.  Some  participants  were  more  interested  in  the  current  problem 
while  others  wanted  to  know  additional  information  regarding  every  aircraft  on  the 
flightline.  Some  participants  wanted  to  see  the  impact  of  their  decisions  on  the  overall 
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flying  schedule  while  others  were  interested  in  information  regarding  resource  location. 
The  diversity  of  interests  and  information  needed  to  make  decisions  necessitates  a  user 
interface  design  tailored  to  different  user  types. 

SSLC2  must  present  the  right  information  at  the  right  time  for  the  user  to  make  an 
actionable  decision  and  allow  the  user  to  filter  out  all  the  unnecessary  information.  Each 
user  wants  the  ability  to  tailor  their  screen  and  views  to  their  specific  job  requirements. 

6.1.5  Information  Playback 

The  current  environment  is  so  manually  labor  intensive,  the  SSLC2  experiment  gave 
many  of  the  participants  a  first  opportunity  to  explore  automated  tracking  of  many  of 
their  tasks  and  decisions.  Using  these  concepts,  participants  expressed  interest  in  being 
able  to  review  what  occurred  over  a  previous  time  period  and  study  the  decisions  and 
impacts  for  trends  analysis  and  lessons  learned. 

6.1.6  Validation  and  Alerts 

As  mentioned  earlier,  Expeditors  currently  use  two-way  communication  through 
radios.  Participants  expressed  the  necessity  to  have  any  potential  system  give  them 
feedback  that  a  message  was  received  and  an  action  or  task  had  started.  Considerations 
include  communicating  flying  schedule  and  resource  changes  to  those  on  the  flightline, 
monitoring  when  tasks  are  complete  and  sending  resources  to  site. 

Participants  also  liked  the  idea  of  visualizing  the  tasks  required  to  complete  a  job  and 
monitoring  the  progress  of  those  tasks.  Suggestions  include  sending  notices  to  resources 
to  work  the  tasks  and  sending  notices  if  task  completion  times  are  in  jeopardy  of 
impacting  the  schedule. 

6.1.7  Fix/Swap  Decision  Recommendations 

SSLC2  should  have  more  complex  algorithms  for  fix/swap  decision  (i.e.  not  just 

based  on  fix  time).  SSLC2  should  investigate  additional  input  variables  for  the  fix/swap 
recommendation.  Expeditors  do  not  work  just  one  fix/swap  decision  or  one  aircraft  at  a 
time  but  are  multi-tasking  many  problems  across  the  flightline.  Expeditors  have  many 
different  parameters  in  their  decision  strategy  and  SSLC2  should  incorporate  these 
complex  variables  and  strategies. 
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6.1.8  Performance  Feedback 

SSLC2  should  have  the  ability  to  include  reports  on  performance  metrics.  This  will 
provide  the  users  feedback  on  how  their  decisions  affect  performance  metrics  over  time. 
The  feedback  to  the  user  shows  the  system  is  providing  useful  support.  This  feedback  can 
be  accomplished  in  two  ways.  Two-way  communication  is  the  most  important  form  of 
feedback  expressed  by  the  participants.  They  need  validation  their  messages  were 
received  and  acted  upon.  The  second  form  of  performance  feedback  supports  metric 
improvements.  With  the  ability  to  playback  past  actions  quickly  they  can  analyze  impacts 
to  their  performance  metrics  and  determine  how  actions  impacted  those  metrics.  In 
essence,  the  can  drill  down  to  the  cause  and  effect  of  changes  to  performance  metrics 
using  the  playback  capabilities  and  make  future  strategy  decisions  based  on  those  results. 

6.2  REQUIREMENTS 

An  output  of  this  research  is  to  provide  requirements  that  can  be  transitioned  into  an 
existing  legacy  system.  Requirement  building  has  taken  place  throughout  the  entire 
program.  They  were  defined  through  the  literature  search,  data  collection,  and  field  test. 

6.3  RESEARCH  QUESTIONS 

The  Spiral  One  effort  identified  SSLC2  research  areas  for  the  future  spirals  as  well  as 
future  research  projects  beyond  the  current  SSLC2  contract  scope.  This  section  presents 
these  research  questions. 

The  main  research  issue  for  the  next  SSLC2  Spiral  is  how  to  integrate  information 
from  enhanced  data  streams  when  focusing  on  the  multiple  tasks  and  problems  that  occur 
on  the  flightline.  Spiral  One  focused  on  one  aircraft  problem,  however,  SSLC2  must  be 
able  to  integrate  information  in  a  complex  environment  in  which  multiple  problems  and 
tasks  are  occurring.  This  research  focus  supports  one  of  General  Martin’s  (AFMC/CC) 
main  focus  areas  as  stated  in  October  2004,  “We  need  to  visualize  our  environment  and 
transfer  all  this  information  into  actionable  decisions.” 

Xj 

Research  should  address  various  ways  the  information  is  integrated  and  presented, 
and  how  various  user  types  would  interact  with  the  system.  Research  questions  include 
which  information  can  be  sensed?  How  should  the  information  be  fused  so  as  not  to 
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overwhelm  the  user?  Who  should  get  what  information,  when,  and  how  should  it  be 
presented?  How  can  the  necessary  information  be  provided  to  the  user  in  a  timely 
manner,  on  a  small  portable  system  that  allows  the  user  to  make  actionable  decisions  and 
maintain  situation  awareness?  Depending  on  the  decisions  to  be  made,  different 
visualization  techniques  may  be  better  for  portraying  specific  information.  The  goal  is  to 
ensure  that  users  are  not  overwhelmed  with  information  they  do  not  need. 

The  ability  to  query  the  information  should  be  studied,  including  investigation  of  the 
use  of  computer  agents  that  can  be  tasked  to  search  and  provide  appropriate  information. 
Computer  agents  that  adapt  and  leam  user  needs  and  preferences  could  be  incorporated  in 
the  future.  External  research  in  this  area  is  progressing  for  searching  and  presenting 
information  from  large  web-based  information  sources. 

Decision  support  is  directly  linked  to  the  user’s  ability  to  maintain  situational 
awareness  (SA),  both  local  SA  (specific  task)  and  global  SA  (entire  situation).  Expeditors 
need  to  maintain  global  SA,  but  also  focus  on  specific  tasks.  How  the  system  can  help  the 
user  maintain  global  SA  when  they  often  must  drill  down  to  specific  problems  must  be 
considered. 

Another  research  focus  would  be  evaluating  the  use  of  a  hybrid  decision  support 
system  using  an  interactive  model  based  approach  versus  a  traditional  decision  support 
tool  that  is  based  only  on  information  fusion  and  presentation.  With  this  approach, 
interactive  algorithms,  for  example  sortie  scheduling  algorithms,  can  be  implemented  to 
include  real  time  human  interaction. 

This  hybrid  decision  support  system  could  include  algorithms  that  determine  the 
assignment  of  resources  for  the  week  while  allowing  the  Expeditor  or  ProSuper  to  change 
those  assignments  based  on  information  not  available  within  the  algorithm.  These 
assignment  changes  are  often  based  on  supervisors’  personal  knowledge  of  capabilities  or 
their  desire  to  balance  training  requirements  and  match  skill  sets,  etc.  Research  could  be 
conducted  to  investigate  whether  this  approach  would  help  the  supervisor  expedite 
decisions  that  are  currently  made  using  manual  information  related  to  personnel 
certifications,  skill  level,  training  records,  appointments,  etc. 
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Another  research  issue  is  how  collaboration  among  multiple  decision  makers  on  the 
flightline  affects  the  decision  making  process  and  how  to  best  support  collaborative 
decision  making.  If  flightline  personnel  (MOC,  ProSuper,  Expeditors,  Crew  Chiefs, 
specialists,  scheduler,  etc)  were  provided  with  collaboration  tool  how  would  their 
decisions  strategies  change?  How  would  the  process  change?  Would  performance 
metrics  be  positively  impacted? 

6.4  SPIRAL  TWO  AND  THREE  PLANS 

The  vision  of  SSLC2  is  to  provide  decision  quality  information  from  multiple  data 
streams  including  real  time  location  and  status  data  for  monitoring  capabilities,  as  well  as 
algorithms  to  equate  resource  availability  to  operational  capability.  Decision-makers  will 
be  provided  insight  to  the  operational  impact  of  their  decisions.  The  SSLC2  effort  is 
researching  the  fusion  of  technology  with  information  to  provide  decision  support  and 
situational  awareness  aimed  at  demonstrating  the  capability  for  improved  decision 
making  that  can  affect  operational  capability. 

Spiral  One  was  scoped  solely  at  wing-level  personnel  focusing  on  one  primary 
decision,  the  fix/swap  decision  of  one  aircraft.  The  goal  was  to  test  the  proof  of  concept 
of  the  fusion  of  RFID/RTLS  technology  to  provide  decision  support  and  situational 
awareness.  The  Scientific  Study  showed  that  the  user  preferred  the  integrated  decision 
support  tool  over  RFID  technology  that  does  not  integrate  information  based  on  the  user 
tasks. 

Spiral  Two  is  an  instantiation  of  the  SSLC2  concept  using  the  VSLRC  contract 
modification  to  SSLC2,  providing  the  Air  Force  Space  Command  (AFSPC)  and  Space 
and  Missile  Systems  Center  (SMC)  an  actual  implementation  of  the  SSLC2  concepts. 
Rather  than  using  RFID/RTLS  technology,  VSLRC  fuses  data  from  numerous  AFSPC 
sources  to  provide  logistics  status  of  space  ground  assets  to  determine  their  operational 
capability.  Spiral  Two  is  scoped  at  the  theater  level  personnel  focusing  on  the  Space 
users,  providing  situational  awareness,  and  making  logistics  support  decisions  to  keep 
operational,  communication,  and  equipment  status  as  high  as  possible.  Spiral  Two 
research  will  focus  on  assessing  user  opinions  of  the  visualizations  and  improvements  to 
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situational  awareness  provided  by  this  new  capability.  The  previous  system  was  manual 
and  non-standard  across  the  space  systems. 

Spiral  Three  will  expand  on  the  work  accomplished  in  Spiral  One  focusing  on  how 
enhanced  data  streams  impact  complex  multiple  decisions.  Collaboration  among  users 
and  integration  of  information  is  a  key  part  of  this  TAD.  Multiple  users  will  have  access 
to  the  same  information.  The  users  will  be  able  to  work  multiple  problems  on  multiple 
aircraft  at  the  same  time.  The  technology  concept  will  be  expanded  beyond  RFID/RTLS 
location  information  to  include  additional  sensor  information.  Research  questions 
include:  How  does  the  visualization  techniques  affect  the  decision  making  process?  Does 
data  fusion  improve  the  timeliness  of  information  extraction  and  decision  making?  How 
is  SA  affected?  A  key  component  of  the  research  will  be  SA,  including  the  evaluation  of 
both  global  (entire  flightline)  and  local  (current  specific  problem)  SA. 


7  Conclusion 

SSLC2  is  investigating  technologies  and  techniques  to  autonomously  collect  and  fuse 
critical  data  to  provide  decision  quality  information  and  effectively  present  information  to 
support  cognitive  tasks  performed  by  logistics  and  operations  decision-makers.  Today’s 
logistics  and  operational  environment  has  little  cross  echelon  situational  awareness.  Data 
capture  and  decision  analysis  are  largely  manual  processes.  Many  of  the  current  systems 
are  standalone  and  do  not  share  data.  Personnel  are  often  challenged  when  trying  to 
locate  the  resources  they  need  to  perform  their  job.  Personnel  need  a  means  to  identify 
the  impact  of  their  logistics  actions  on  operational  capability.  SSLC2  provides  decision 
quality  information  fused  from  enhanced  data  streams  such  as  RFID/RTLS  technologies, 
smart  sensors,  computing  technologies,  and  information  networks. 

SSLC2  Spiral  One  focused  on  developing  requirement  specifications  for  using  real 
time  sensing  information  as  well  as  legacy  system  data  to  help  flightline  personnel  make 
decisions  and  understand  how  their  decisions  are  affecting  flightline  operations.  A 
Scientific  Study  was  designed  and  conducted  to  study  the  impacts  of  enhanced  data 
streams  provided  by  RFID/RTLS  technology  and  fused  data  from  legacy  systems  and 
potential  future  automated  capabilities.  The  Scientific  Study  objectives  were  to: 
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1) 


Validate  the  cognitive  model  developed  from  data  collection  interviews 
through  verbal  protocol  (source)  techniques  and  to  determine  what 
information  is  most  useful  to  help  Expeditors  make  flightline  decisions. 

2)  Evaluate  user  performance  and  opinions  on  enhanced  data  streams. 

The  Scientific  Study  compared  three  conditions.  The  SSLC2  condition  integrated 
RFID/RTLS  technology  with  flightline  information  for  improved  decision  making.  The 
off-the-shelf  technology  condition  called  WhereNet  provided  location  of  resources  but 
did  not  integrate  the  information  with  the  Expeditor’s  work  and  decision  process.  The 
Baseline  condition  was  included  as  a  control  condition,  allowing  for  comparison  of  the 
SSLC2  approach  to  current  practice  and  enabling  validation  of  the  cognitive  model. 

The  Scientific  Study  focused  on  the  fix  or  swap  decision  construct  identified  in  the 
data  collection  and  cognitive  task  analysis  processes.  The  verbal  protocol  technique  was 
used  to  collect  data  related  to  the  types  of  information  and  processes  the  Expeditors  use 
to  make  the  fix/swap  decision.  Participants  were  also  asked  to  provide  their  opinions  of 
the  three  conditions.  The  hypothesis  was  that  participants  would  prefer  the  SSLC2 
approach  compared  to  the  WhereNet  off-the-shelf  RFID/RTLS  technology  system. 
Because  participants  are  so  familiar  with  current  practice  and  often  resistant  to  change,  it 
was  hypothesized  that  there  would  be  no  difference  in  preferences  between  the  SSLC2 
approach  and  the  Baseline  current  practice  approach. 

Results  showed  that  the  hypothesis  that  participants  would  prefer  the  SSLC2 
approach  compared  to  the  WhereNet  and  Baseline  was  supported.  Participants  gave 
higher  preference  rankings  to  SSLC2  over  both  WhereNet  and  Baseline.  WhereNet  was 
ranked  lowest,  but  it  was  not  statistically  different  from  Baseline.  When  participants  rated 
their  agreement  with  a  variety  of  task  statements  there  were  no  differences  between 
SSLC2  and  WhereNet.  User  commented  that  interaction  with  WhereNet  was  somewhat 
difficult.  The  results  also  showed  the  cognitive  model,  developed  after  initial  interviews, 
was  accurate.  The  Scientific  Study  verbal  protocol  results  provided  more  detailed 
decision  strategies. 

The  results  also  indicate  that  SSLC2  must  consider  how  enhanced  data  streams  (such 
as  location  of  resources)  can  be  integrated  with  existing  information  to  support  user  tasks 
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and  decisions  within  the  context  of  their  work.  Location  information  as  a  stand  alone 
system  (WhereNet)  was  not  preferred. 

Participant  feedback  indicated  that  they  supported  the  concept  of  enhanced  data 
streams  using  RFID/RTLS  technology  and  provided  feedback  and  suggestions  for 
additional  information  that  can  be  sensed. 

The  next  step  for  SSLC2  is  to  evaluate  the  integration  of  information  from  enhanced 
data  streams  when  focusing  on  the  multiple  tasks  and  problems  that  occur  on  the 
flightline. 

I 
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Appendix  A 
SSLC2  Acronyms 
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Acronym 

Definition 

A/C 

Aircraft 

AB 

After  Burner 

ACS 

Agile  Combat  Support 

AFB 

Air  Force  Base 

AFFOR 

Air  Force  Forces 

AFI 

Air  Force  Instruction 

AFMC 

Air  Force  Materiel  Command 

AFRC 

Air  Force  Reserve  Command 

AFRES 

Headquarters  Air  Force  Reserve 

AFRL 

Air  Force  Research  Laboratory 

AFRL/HEAL 

Air  Force  Research  Laboratory  Warfighter  Readiness  Research 

Division 

AFRL/HECV 

Air  Force  Research  Laboratory  Information  Visualization 

AFSC 

Air  Force  Specialty  Code 

AFSOC 

Air  Force  Special  Operations  Command 

AFTO 

Air  Force  Technical  Order 

AGE 

Aerospace  Ground  Equipment 

AIT 

Automatic  Information  Technologies 

ALS 

Autonomic  Logistics  System 

AMU 

Aircraft  Maintenance  Unit 

ANG 

Air  National  Guard 

ANOVA 

Analysis  of  Variance 

AOC 

Air  and  Space  Operations  Center 

API 

Application  Programming  Interface 

AR 

Aerospace  Repair 

ATD 

Advanced  Technology  Demonstration 

ATO 

Air  Tasking  Order 

ATOC 

Air  Terminal  Operations  Center 

C2 

Operations  Command  and  Control 
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Acronym 

Definition 

CAMS 

Core  Automated  Maintenance  System 

CAP 

Combat  Air  Patrol 

CONUS 

Continental  United  States 

DSRC 

Dedicated  Short  Range  Communication 

E&E 

Environmental  &  Electrics 

EMOC 

Enhanced  Maintenance  Operations  Center 

ETIC 

Estimated  Time  to  Completion 

FMC 

Fully  Mission  Capable 

FOD 

Foreign  Object  Damage 

FSE 

Flying  Schedule  Effectiveness 

G081 

Air  Mobility  Command  CAMS 

GUI 

Graphical  User  Interface 

HQ 

Headquarters 

hup 

HyperText  Transport  Protocol 

ICT 

Integrated  Combat  Turns 

IMIS 

Integrated  Maintenance  Information  Systems 

IRA 

Interface  Requirements  Agreements 

ASO 

Inspection  Operations 

ITV 

In-Transit  Visibility 

JDBC 

Java  Database  Connectivity 

JOAP 

Joint  Oil  Analysis  Program 

JQS 

Job  Qualification  Standard 

KPP 

Key  Performance  Parameters 

LCOM 

Logistics  Composite  Model 

LD/HD 

Low  Density/High  Demand 

LG 

Logistics  Group 

LOCIS 

Logistics  Command  and  Control  Information  Support 

MAJCOM 

Major  Command 

MC 

Mission  Capable 

MEDAF 

Mediterranean  Air  Force 
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Acronym 

Definition 

MICAP 

Mission  Capability 

MOC 

Maintenance  Operations  Center 

NMC 

Non-Mission  Capable 

OFM 

Operator  Function  Model 

OFM-COG 

Operator  Function  Models  supplemented  with  COGnitive 

Information 

OG 

Operations  Group 

PMC 

Partially  Mission  Capable 

ProSuper 

Production  Supervisor 

RAM 

Random  Access  Memory 

RF 

Radio  Frequency 

RFID 

Radio  Frequency  Identification 

RTLS 

Real  Time  Locating  System 

SA 

Situational  Awareness 

SME 

Subject  Matter  Expert 

SSLC2 

Smart  Systems  for  Logistics  Command  and  Control 

STABOS 

Stability  Operations 

TAD 

Technology  Availability  Date 

TO 

Technical  Orders 

TRT 

Take-Off  Rated  Thrust 

USMEDCOM 

United  States  Mediterranean  Command 

VOIP 

Voice  over  Information  Protocol 

VSLRC 

Virtual  Space  Logistics  Readiness  Center 
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1  Introduction 

Appendix  B,  Data  Collection,  contains  background  information  detailing  the  data 
collection  and  data  analysis  processes  highlighted  in  the  SSLC2  Spiral  One  Final  Report. 
This  document  follows  the  main  document  giving  additional  details  about  the  Spiral  One 
Methods  and  Approach  including  systems  analysis,  data  collection,  storyboard 
development,  and  knowledge  representations.  The  Results  and  Discussion  section  contain 
the  original  data  rankings  and  frequencies  and  information  obtained  through  verbal 
protocol  analysis. 

2  Spiral  One  Methods  and  Approach 

2.1  SYSTEMS  ANALYSIS  AND  DATA  COLLECTION 

A  variety  of  Data  Collection  Strategies  were  used  throughout  the  Spiral  One 
Scientific  Study  design  and  development  to  obtain  user  feedback  on  decision  making. 
Data  collection  revealed  that  some  potential  users  make  a  variety  decisions  and, 
therefore,  multiple  techniques  to  gather  data  were  required. 

Through  process  interviews,  cognitive  interviews,  and  shadowing  the  team  collected 
data  to  help  understand  the  decision  making  process  and  daily  decisions  that  potential 
users  make.  This  data  was  also  used  to  refine  requirements  for  the  Scientific  Study. 
Figure  1  shows  the  different  data  collection  methods  used  at  each  location  during  early 
data  collection  efforts. 
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Data  Collection  Method  by  Location 
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Cognitive 
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Process 
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Process, 

Cognitive 

Questions 

Deployment 

Managers 
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Cognitive 
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Process, 

Cognitive 

Interviews 

Figure  1:  Data  Collection  Method  by  location 


Data  collection  revealed  a  number  of  user  problems.  The  problems  in  Table  1  are 
common  flightline  problems  addressed  by  flightline  maintainers. 


Table  1:  User  Problems  and  Issues 


User  Problem 

Issues 

Tracking  and  allocating 
borrowed  equipment  across 
Combat  AGE  Teams 
(CATS). 

o  Which  CAT  has  a  surplus  of  a  particular  piece  of  equipment 
that  can  be  borrowed? 
o  Where  is  the  equipment  located? 
o  What  is  the  operational  status  of  the  equipment? 
o  What  is  the  availability  of  the  equipment? 
o  When  should  the  borrowed  equipment  be  returned? 

Identify  the  “right"  equipment 
for  a  particular  maintenance 
task. 

o  What  type  of  equipment  is  needed  for  the  maintenance  task 
and  what  is  the  estimated  duration  for  the  task? 
o  Where  is  the  equipment  located? 
o  What  is  the  operational  status  of  the  equipment? 
o  What  is  the  availability  of  the  equipment? 

Identify  which  equipment 
technicians  are  finished 
using  and  where  it  should 
go. 

o  What  visual  cues  indicate  that  a  technician  is  finished  using  a 
piece  of  equipment? 

o  Where  is  the  equipment  required  next? 

Identify  the  shortfalls  in 
available  equipment  due  to 
inspections/preventative 

o  What  equipment  needs  to  be  pulled? 
o  What  equipment  is  going  to  replace  it  and  can  it  be  used  until 
the  pulled  equipment  is  available? 
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User  Problem 

Issues 

maintenance  tasks. 

Identify  the  equipment  to  be 
brought  in  for 
inspections/preventive 
maintenance. 

o  What  equipment  is  overdue  or  coming  due? 
o  What  inspections  can  be  supported  by  available  personnel? 
o  Will  equipment  maintenance  have  a  negative  impact  on  the 
flying  schedule  or  aircraft  schedule  maintenance? 

Equipment  is  not  delivered  in 
a  timely  manner  after  call  ins 
due  to  lack  of  prioritization. 

o  Prioritization  is  based  upon  multiple  disparate  factors  (time  of 
request,  flying  schedule,  and  identifying  the  “right”  equipment), 
o  What  is  the  average  delivery  time? 
o  Is  there  enough  equipment  available  or  will  it  need  to  be 
borrowed? 

Identify  the  “right”  equipment 
for  a  deployment. 

o  What  type  of  equipment  is  needed  for  the  deployment? 
o  What  equipment  is  the  healthiest  (up  to  date  on  their 
inspections)? 

o  Where  is  the  equipment  located? 

Initial  data  collection  also  highlighted  critical  sortie  production  resources  that  would 
provide  value  if  tagged  with  RFID/RTLS  capabilities.  All  of  these  resources  are  low 
density,  high  demand  assets.  Many  of  these  items  are  either  specialized  or  shared 
between  units  according  to  Luke  AFB  interviews.  These  resources  include: 

•  Mules 

•  Dash  60s 

•  Nitrogen  Carts 

•  Oxygen  Carts 

•  Coleman  Towbars 

•  Stands 

•  Jacks 

The  table  below  (Table  2)  was  created  to  depict  how  different  roles  within  each  data 
collection  site  assumed  different  functional/decision  making  responsibility.  The  left  side 
is  representative  of  three  different  functions  (transportation,  dispatch,  and  controller)  as 
use  case  actors  within  the  September  2004  SSLC2  use  cases.  Each  role  was  aligned  with 
a  function  in  relation  to  that  actor  in  the  use  cases.  The  columns  then  identify  the  specific 
site  role  who  performs  those  functions. 
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Table  2:  Use  Case  Actor  Roles 


Use  Case  Actor 

F  Luke  AFB  S 

CAT  Driver 

[  CAT  Lead 

AGE  Pro  Super 

|AGE  Chief  | 

AGE  Transport  Function 

X 

■ 

Muarag-fsEst 

mmm 

AGE  Dispatch  Function 

X 

X 

AGE  Control  Function 

X 

X 

X 

Use  Case  Actor 

445th 

AGE  Driver 

AGE  Chief 

AGE  Transport  Function 

X 

AGE  Dispatch  Function 

X 

I 

AGE  Control  Function 

X 

X 

Use  Case  Actor 

Sprinqfield 

j 

AGE  Driver 

AGE  Chief 

i 

AGE  Transport  Function 

X 

i 

AGE  Dispatch  Function 

X 

AGE  Control  Function 

X 

X 

1 

Table  3  represents  data  collected  through  site  visits  related  to  the  subject  data 
required  to  feed  Smart  Systems.  The  left  sides  of  each  site  table  show  the  use  case  actor 
which  represents  the  respective  system.  The  columns  along  the  top  show  the  actual 
physical  system  that  will  feed  that  respective  data  to  Smart  Systems.  The  systems  differ 
from  site  to  site  and  sometimes  are  merely  a  spreadsheet  or  not  even  existent. 


Table  3:  Use  Case  System  actors 


445th  WPAFB 

Use  Case  System  Actor 

GOBI 

CAMS 

Local  Excel 
Spreadsheets 

Local  C2 
System 

Non-Existant 

Information 

System 

Maintenance  Repository 

X 

X 

Smarl  Sensors  (RFID) 

SSLC2  Data  Store 

X 

Technical  Data  Repository 

■i 

X 

X 

Luke  AFB 

Use  Case  System  Actor 

GOBI 

CAMS 

Local  Excel 
Spreadsheets 

Local  C2 
System 

NorvExistant 

Information 

System 

Maintenance  Repository 

X 

X 

Smart  Sensors  (RFID) 

X 

kfd  1  *rl 

X 

Technical  Data  Repository 

X 

MOC  Status  Repository 

Springfield  ANG.  OH 

Use  Case  System  Actor 

G081 

CAMS 

Local  Excel 
Spreadsheets 

Local  C2 
System 

Non-Existant 

Information 

System 

Maintenance  Repository 

X 

X 

Smart  Sensors  (RFID) 

X 

SSLC2  Data  Store 

X 

Technical  Data  Repository 

X 

MOC  Status  Repository 

X 

*  ASB  -  Aircraft  Status  Board:  Local  application  tracking  sortie  generation  information 

**  C2DB  -  Command  and  Control  Database:  Local  MSAccess  database  tracking  sortie  generation  information 


2.2  STORYBOARD  DEVELOPMENT 

Leveraging  information  captured  during  data  collection  and  knowledge  representation 
stages,  a  series  of  concepts  and  storyboards  were  created  and  used  as  input  into  the 
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Graphical  User  Interface  (GUI)  design  as  well  as  the  software  use  cases  for  development 
of  the  simulation  test  system.  Figures  2,  3,  and  4  are  some  of  the  preliminary  screens 
designs  created  from  an  equipment  manager  work  perspective. 
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Figure  2:  Smart  Systems  Initial  To-Do  Design 
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Figure  3:  Smart  Systems  Initial  Equipment  and  Delivery  Schedule 
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Figure  4:  Smart  Systems  Initial  Overview  Design 


2.3  KNOWLEDGE  REPRESENTATIONS 
2.3.1  Hierarchical  Task  Analysis 

The  hierarchical  task  analysis  diagrams  (Figures  5-10)  illustrate  the  Expeditor’s  tasks  at  a 
very  high  level. 


Ill 


1  1  3 


Figure  6:  Monitor  Flightline  Task  Breakdown  (Continues  Next  Page) 


Monitor  Flightline 
Cont. 
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Figure  7:  Monitor  Flightline  Task  Breakdown  (Continued) 


Monitor  A/C  Landing 
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Figure  8:  Monitor  Aircraft  Landing  Task  Breakdown 
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Figure  9:  Monitor  Aircraft  Launch  Task  Breakdown 
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Figure  10:  Assess  Aircraft  Status  Task  Breakdown 


2.3.2  Operator  Function  Models 

An  Operator  Function  Model  (OFM)  supplemented  with  COGnitive  information 
(OFM-COG)  was  created  to  present  the  Expeditor’s  Cognitive  Model  [10].  For  the  OFM 
5  sub-functions  of  the  Monitor  Flightline  Maintenance  were  identified:  Monitor  aircraft 
launch,  Monitor  aircraft  landing,  Assign  aircraft  to  sorties,  Assess  aircraft  status,  and 
Evaluate  fleet  health  (Figures  11-14).  Tables  5-10  illustrate  the  OFM  Cognitive  Task 
Analysis.  Information  related  to  evaluating  fleet  health  was  not  collected.  Table  4 
identifies  the  definitions  for  the  Task  agents  used  in  this  model. 


Table  4:  Millers  Cognitive  Task  Transactions 


Cognitive  Task  Agent 

Definition 

Human  Information 
Processing  Resources 

Adapt/Learn 

Making  and  remembering  new 
responses  to  a  learned  situation 

Long  term  memory 

Categorize 

Defining  and  naming  a  group  of 
things 

Perceptual  sensitivity,  Long 
term  memory 

Code 

Translating  the  same  thing  from 
one  form  to  another 

Long  term  memory,  Working 
memory,  Response  precision 

Compute 

Figuring  out  a  logical  or 
mathematical  answer  to  a 
defined  problem 

Working  memory,  Processing 
strategies 

Control 

Changing  an  action  according  to 
plan 

Response  precision 

Count 

Keeping  track  of  how  many 

Sustained  attention,  Working 
memory, 

Decide/select 

Choosing  a  response  to  fit  the 
situation 

Long  term  memory,  Processing 
strategies 

Detect 

Is  something  there? 

Perceptual  sensitivity, 

Distributed  attention 

Display 

Showing  something  that  makes 
sense 

Response  precision 

Edit 

Arranging  or  correcting  things 
according  to  rules 

Selective  attention,  Long  term 
memory, 

Filter 

Straining  out  what  does  not 
matters 

Selective  attention 

Goal  image 

A  picture  of  a  task  well  done 

Long  term  memory,  Processing 
strategies 

Identify 

What  is  it  and  what  is  its  name? 

Perceptual  discrimination,  Long 
term  memory,  Working 
memory, 
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Cognitive  Task  Agent 

Definition 

Human  Information 
Processing  Resources 

Input  Select 

Select  something  to  pay  attention 
to  or  next 

Selective  attention,  Perceptual 
sensitivity, 

Interpret 

What  does  it  mean? 

Sustained  attention,  Long  term 
memory, 

Message 

A  collection  of  symbols  sent  as  a 
meaningful  statement 

Response  precision 

Plan 

Matching  resources  in  time  to 
expectations 

Working  memory,  Processing 
strategies 

Purge 

Getting  rid  of  the  irrelevant  data 

Selective  attention 

Queue  to  channel 

Lining  up  to  process  in  the  future 

Working  memory,  Processing 
strategies 

Reset 

Getting  ready  from  some 
different  action 

Selective  attention,  Response 
precision, 

Search 

Looking  for  something 

Perceptual  sensitivity, 

Sustained  attention, 

Store 

Keeping  something  intact  for 
future  use 

Long  term  memory,  Working 
memory 

Store  in  buffer 

Holding  something  temporarily 

Working  memory,  Processing 
strategies 

Test 

Is  it  what  it  should  be? 

Perceptual  sensitivity,  Long 
term  memory,  Working  memory 

Transmit 

Moving  something  from  one 
place  to  another 

Response  precision 
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1 .  Aircraft  to  launch 

2.  Deviation  occurred 

3.  Launch  began 

4.  Crew  signals  ready 

5.  Partial  crew  arrived 

6.  Full  crew  arrived 

7.  Pilot  starts  engine 

8.  Blocks  removed 

9.  A/C  take-off 

10.  Full  crew  arrived 

Figure  11:  Monitor  Aircraft  Launch  OFM 
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1 .  Aircraft  is  landing 

2.  Squawk  received 

3.  Identified  code 

4.  Safety  and  rules  considered 

5.  Parking  location  identified 

6.  Notify  correct  personnel 

Figure  12:  Monitor  Aircraft  Landing  OFM 
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Monitor  Flightline 
Maintenance 


/Assign/identifyN  /Assign/identifyN  /Assign/identify' 
v  A 1C  as  FMC  )  V  A/C  as  PMC  /  V  A/C  as  NMC  y 


/^Keceive"^. 
(Personnel  report) 
X^of  problem^/ 


Review  781 
forms 


'Review  Status^ 
v  Sheet  y 


''Attend  Status^ 
.  Meeting  . 


Review^ 

scheduled 

maintanence 


Determine  Fueh 
level  > 


Jetermine  LOX 
level 


(Assess  Weather) 


xDetermine^s 

maintenance 

s^stage 


(identify  Problem} 


1 .  Aircraft  status 

2.  Monitor/review/receive  information 

3.  Problem  reported 

4.  Problem  requires  external  specialist 
assessment. 

5.  Problem  identified 

6.  Execute  decision 

7.  Implement  fix  procedures 

8.  Resources  required 

9.  Resources  identified 

10.  Resources  available  via  other  units 
and/or  owners. 

1 1 .  Resources  need  pickup 

12.  Resources  available 

13.  Progress  evaluated 

14.  Status  updated 


Update  status^ 
.  sheet  > 


Request 

Assessment 


Begin  Fix 
Procedures 


t)etermine  how\ 
.to  meet  sortie  ) 


Begin  Swap 
procedures 


Begin  Spare 
procedures 


Determine 

resources 


(Monitor  progress) 


Locate 

Resources 


Call  for 
Resources 


Pickup 

Resources 


Figure  13:  Assess  Aircraft  Status  OFM 
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Monitor  Flightline 
Maintenance 


2.  Sortie  requested/scheduled 

3.  Aircraft  status  identified 

Figure  14:  Assign  Aircraft  to  Sortie  OFM 


Table  5:  "Monitor  Aircraft  Launch"  function  OFM-COG  Analysis 
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issues,  current  state 
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3  Scientific  Study  Results  and  Discussion 


3.1  SPIRAL  ONE  POST  TEST  RANKING  DECISION  AND  INFORMATION  TABLES 

Participants  were  asked  to  complete  questionnaires  at  the  end  of  the  Scientific  Study 
to  obtain  feedback  on  information  helpful  in  making  flightline  decisions.  Below  are 
detailed  frequency  tables  which  are  summarized  in  the  main  document. 

3.1.1  Decision  Elements  Rankings 

Post  study  questionnaire  Part  2  requested  participants  to  assign  a  rating  indicating  the 
importance  of  certain  information  on  decisions.  In  the  first  half  of  the  questionnaire 
participants  rated  7  information  elements  (1-not  at  all  important  to  7-extremely 
important)  on  their  fix/swap  decision.  The  detailed  frequency  data  is  displayed  in  Tables 
9-12. 


Table  9:  Decisions  that  Impact  FSE,  AA  and  MC  Rates 


1  Not  at  all 
Important 

2  Somewhat 
Important 

3  Slightly 
Unimportant 

4  Neutral 

5  Slightly 
Important 

6  Somewhat 
Important 

7  Extremely 
Important 

Fix  or  Swap? 

1 

1 

4 

8 

How  should  A/C  be 
assigned  for  the  week? 

1 

6 

6 

1 

How  should  today's  flying 
schedule  be  changed? 

1 

3 

2 

How  will  aircraft  be 
assigned  for  the  month? 

1 

4 

7 

1 

Should  we  CANN? 

2 

1 

6 

bh 

Should  the  flight  be 
delayed? 

2 

5 

mm 

What  personnel  should  be 
assigned  to  maintenance 
tasks? 

1 

4 

5 

IH 

Which  aircraft  should  be 
assigned  to  which  sortie? 

1 

1 

6 

6 

Table  10:  Information  Influencing  tiie  Fix  or  Swap  Decision 


Information  Influencing  the  Fix  or  Swap  Decision  -  Airlift  and  Fighter  Combined 

1  Not  at  all 
Important 

2  Somewhat 
Unimportant 

3  Slightly 
Important 

4  Neutral 

5  Slightly 
Important 

6  Somewhat 
Important 

7 

Extremely 

Important 

A/C  Availability  Metric 

1 

2 

2 

7 
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Table  1 1 :  information  Influencing  the  Fix  or  Swap  Decision  -  Airlift  Only 


information  Influencing  the  Fix  or  Swap  Decision  -  Airlift  Onl 


1  Not  at  all  2  Somewhat  3  Slightly 
Important  Unimportant  Important 


A/C  Availability  Metric 


Flying  Schedule 
Effectiveness  (FSE) 
Rate 


MC  Rate 


Monthly  Flying  Schedule 


Today's  Flying  Schedule 


Tomorrow's  Flying 
Schedule 


Weekly  Flying  Schedule 


5  Slightly  6  Somewhat  7  Extremely 
Important  Important  Important 


Table  12:  Information  Influencing  the  Fix  or  Swap  Decision  -  Fighter  Only 


3.1.2  Data  Elements  Frequency  Tables 

The  Scientific  Study  questionnaires  asked  participants  to  rate  on  a  scale  of  1  to  7,  the 

importance  of  data  elements  that  would  be  available  in  an  electronic  decision  support 
tool.  The  data  elements  were  broken  into  five  categories:  Aircraft,  Personnel,  AGE, 
Supply,  and  Other.  They  were  also  asked  to  write  in  any  data  elements  they  considered 
important.  Tables  13-17  provide  the  detailed  frequency  information  related  to  these  data 
elements. 


Table  13:  Important  Aircraft  Data  Elements 


Important  Aircraft  Data  Elements 

1  Not  at  all 
Important 

2  Somewhat 
Important 

3  Slightly 
Important 

4  Neutral 

5  Slightly 
Important 

6  Somewhat 
Important 

7  Extremely 
Important 

781  Discrepancies 

1 

5 

8 

Aircraft  Configuration 

3 

9 

Aircraft  Hours 

1 

1 

2 

3 

6 

1 

Aircraft  Maintenance 
History 

2 

1 

4 

2 

4 

1 

CAMS  380  List  of  Jobs 

1 

3 

7 

2 

Current  status  of  Aircraft 
(FMC,  NMC,  PMC,  etc.) 

14 

Engine  Hours  and 
Cycles 

1 

2 

3 

2 

4 

1 

1 

Location  of  Aircraft 

1 

2 

11 

Monthly  Aircraft 
Scheduled  Maintenance 

1 

2 

5 

4 

2 

Monthly  Flying  Schedule 

1 

3 

8 

2 

Time  since  last 
Phase/ISO  Inspection 

1 

1 

2 

2 

4 

2 

2 

Weekly  Aircraft 
Schedule  Maintenance 

2 

1 

8 

3 

Weekly  Flying  Schedule 

2 

8 

4 

Table  14:  Important  AGE  Data  Elements 


1  Not  at  all 
Important 

2  Somewhat 
Important 

3  Slightly 
Unimportant 

4  Neutral 

5  Slightly 
Important 

6  Somewhat 
Important 

7  Extremely 
Important 

107  Requests 

1 

1 

3 

2 

1 

3 

Battery  Level  on 
Equipment 

1 

2 

3 

3 

4 

1 

Fuel  Level  on 
Powered  Equipment 

1 

3 

2 

7 

Level  of  Liquid 
Oxygen 

1 

7 

HBH 

Location  of  AGE  and 
other  equipment 

1 

1 

1 

5 

6 
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1  Not  at  ati 

2  Somewhat 

3  Slightly 

4  Neutral 

5  Slightly 

6  Somewhat 

7  Extremely 

Important 

Important 

Unimportant 

Important 

Important 

Important 

Status  of  AGE 
Equipment 
(FMC/NMC) 


1 


1 


5 


Table  15:  Important  Supply  Data  Elements 


1  Not  at  all 
Important 

2  Somewhat 
Important 

3  Slightly 
Unimportant 

4  Neutral 

5  Slightly 
Important 

6  Somewhat 
Important 

7  Extremely 
Important 

350  Tag 

2 

4 

5 

1 

Bench  Stock 

3 

7 

Location  of  MICAP 
Supply 

10 

Location  of  Supplies 

1 

1 

9 

Status  of  MICAP 
Supply 

2 

4 

n 

Status  of  Supplies 

1 

3 

3 

1 

Table  16:  Important  Personnel  Data  Elements 


Important  Personnel  Data  Elements 

1  Not  at  all 
Important 

2  Somewhat 
Important 

3  Slightly 
Unimportant 

4  Neutral 

5  Slightly 
Important 

6  Somewhat 
Important 

7  Extremely 
Important 

Location  of  Personnel 
Resources 

4 

9 

Monthly  Personnel 
Schedules 

2 

4 

6 

2 

Personnel 

Qualifications 

Specialty 

2 

4 

8 

Personnel 

Qualifications 

Certifications 

1 

3 

10 

Personnel 

Qualifications  Ranks 

1 

1 

2 

5 

5 

Personnel 
Qualifications  Skill 
Level  (3.5.7) 

1 

2 

11 

Personnel  Status 

7 

6 

Special  Certification 
Rosters  (Red  X,  ER, 
intake) 

2 

12 

Weekly  Personnel 
Schedules 

3 

2 

8 

1 

Table  17:  Other  Important  Data  Elements 


Other  Important  Data  Elements 

1  Not  at  all 
Important 

2  Somewhat 
Important 

3  Slightly 
Unimportant 

4  Neutral 

5  Slightly 
Important 

6  Somewhat 
Important 

7  Extremely 
Important 

Squadron  Unit 
monthly  analysis 
indicators  report 

1 

1 

6 

1 

3 

Weather 

1 

3 

10 
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3.2  VERBAL  PROTOCOL  ANALYSIS 

Participants  were  asked  to  think  aloud  during  each  Scientific  Study  scenario  as  a 
first  step  in  the  verbal  protocol  analysis  as  explained  in  Section  5.6  of  the  SSLC2  Final 
Report.  Their  verbal  statements  were  recorded  and  notes  were  taken  in  real  time.  The 
verbal  information  for  fifteen  subjects  were  evaluated,  across  all  three  conditions 
resulting  in  analysis  of  45  scenarios.  Table  1 8  below  is  an  example  of  the  data  collection 
template.  This  example  is  from  the  WhereNet  condition  during  the  fix/swap  decision. 
The  data  were  analyzed  in  several  different  ways.  First  the  information  was  broken  into 
steps.  For  each  step,  the  verbal  information  was  dissected  into  six  categories.  The  first 
category  is  Information.  This  category  refers  to  the  type  of  information  the  participant 
was  seeking,  distributing  or  working  with  during  the  step.  The  second  category  is 
Source,  which  refers  to  where  they  were  seeking  the  information.  The  third  category  is 
Destination,  referring  to  where  the  participant  sent  or  distributed  the  information.  The 
fourth  category  is  Decision  which  indicates  what  decision  they  were  working  on.  The 
fifth  category  is  Time,  which  was  used  to  input  the  time  to  site  or  time  to  fix  information 
that  participant  provided  during  the  scenario.  The  sixth  category  is  Process  which  is  a 
description  of  steps  or  the  process  the  participants  were  taking. 
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Table  18:  Verbal  Protocol  Data  Collection  Example 


3.2.1  Information  Frequency 

The  types  of  information  the  participants  used  during  the  task  were  determined  and  a 
frequency  count  was  tabulated  showing  how  often  the  information  was  referred  to  or  used 
across  all  fifteen  subjects  and  three  conditions  (45  scenarios).  Table  19  provides  the 
frequency  count  data  and  categories  of  information.  The  data  helps  to  determine  the  type 
of  information  the  participants  needed  to  complete  the  tasks. 


Table  19:  Information  Frequency  Table 


Condition 

Information 

Baseline  Experiment 

Baseline  Experiment 

A/C  Availability 

Baseline  Experiment 

A/C  Configuration 

Baseline  Experiment 

A/C  Parking 

Baseline  Experiment 

A/C  Problem 

Baseline  Experiment 

A/C  Status 

Baseline  Experiment 

Current  Time 

Baseline  Experiment 

Entry  Control  Point 

Baseline  Experiment 

Equipment  Allocation 

Baseline  Experiment 

Equipment  Availability 

Baseline  Experiment 

Equipment  Location 

Baseline  Experiment 

ETIC 

Baseline  Experiment 

Flightline  Status 

Baseline  Experiment 

Fuel 

Baseline  Experiment 

Hangar  Status 

Baseline  Experiment 

Job  Number 

Baseline  Experiment 

Mission 

Baseline  Experiment 

Part  Availability 

Baseline  Experiment 

Personnel  Allocation 

Baseline  Experiment 

Personnel  Availability 

Baseline  Experiment 

Personnel  Location 

Baseline  Experiment 

Personnel  Qualifications 

Baseline  Experiment 

Protocol 

Baseline  Experiment 

Resource  Allocation 

Baseline  Experiment 

Resource  Availability 

Baseline  Experiment 

Resource  Problem 

Baseline  Experiment 

Schedule  Change 

Baseline  Experiment 

Swap 

Baseline  Experiment 

T/O  time 

Condition 

Information 

Count  Of 
Information 

Baseline  Experiment 

Task  Assignment 

12 

Baseline  Experiment 

Task  Sequence 

3 

Baseline  Experiment 

Task  Time 

4 

Baseline  Experiment 

Time  to  Fix 

21 

Baseline  Experiment 

Time  to  Site 

31 

Baseline  Experiment 

Tow  Clearance 

1 

Baseline  Experiment 

Weather 

2 

Smart  Systems  Experiment 

0 

Smart  Systems  Experiment 

A/C  Availability 

4 

Smart  Systems  Experiment 

A/C  Configuration 

1 

Smart  Systems  Experiment 

A/C  Problem 

9 

Smart  Systems  Experiment 

A/C  Status 

6 

Smart  Systems  Experiment 

Current  Task 

1 

Smart  Systems  Experiment 

Current  Time 

2 

Smart  Systems  Experiment 

Equipment  Allocation 

2 

Smart  Systems  Experiment 

Equipment  Availability 

2 

Smart  Systems  Experiment 

Equipment  Location 

15 

Smart  Systems  Experiment 

ETIC 

3 

Smart  Systems  Experiment 

Flight  Schedule 

2 

Smart  Systems  Experiment 

Flightline  Status 

8 

Smart  Systems  Experiment 

Fuel 

2 

Smart  Systems  Experiment 

Mission 

1 

Smart  Systems  Experiment 

Part  Availability 

1 

Smart  Systems  Experiment 

Personnel  Allocation 

2 

Smart  Systems  Experiment 

Personnel  Availability 

6 

Smart  Systems  Experiment 

Personnel  Location 

12 

Smart  Systems  Experiment 

Resource  Allocation 

1 

Smart  Systems  Experiment 

Resource  Location 

1 

Smart  Systems  Experiment 

Resources  needed 

1 

Smart  Systems  Experiment 

Swap 

8 

Smart  Systems  Experiment 

T/O  Time 

6 

Smart  Systems  Experiment 

Task  Sequence 

1 

Smart  Systems  Experiment 

Time  to  Fix 

18 

Smart  Systems  Experiment 

Time  to  Site 

32 

Smart  Systems  Experiment 

Time  to  Swap 

5 

Smart  Systems  Experiment 

Weather 

2 

WhereNet  Experiment 

0 

WhereNet  Experiment 

A/C  Availability 

3 

WhereNet  Experiment 

A/C  Configuration 

1 
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Condition 

Information 

Count  Of 
Information 

WhereNet  Experiment 

A 1C  Parking 

1 

WhereNet  Experiment 

A/C  Problem 

5 

WhereNet  Experiment 

A/C  Status 

8 

WhereNet  Experiment 

Coordination 

1 

WhereNet  Experiment 

Crew  Status 

1 

WhereNet  Experiment 

Current  Task 

1 

WhereNet  Experiment 

Current  Time 

5 

WhereNet  Experiment 

Entry  Control  Point  Location 

1 

WhereNet  Experiment 

Equipment  Allocation 

15 

WhereNet  Experiment 

Equipment  Availability 

6 

WhereNet  Experiment 

Equipment  Location 

24 

WhereNet  Experiment 

Equipment  Movement 

1 

WhereNet  Experiment 

ETIC 

10 

WhereNet  Experiment 

Flight  Schedule 

1 

WhereNet  Experiment 

Fuel 

4 

WhereNet  Experiment 

Hangar  Status 

1 

WhereNet  Experiment 

Job  Control  Number 

2 

WhereNet  Experiment 

Part  Availability 

2 

WhereNet  Experiment 

Personnel  Allocation 

15 

WhereNet  Experiment 

Personnel  Availability 

5 

WhereNet  Experiment 

Personnel  Location 

18 

WhereNet  Experiment 

Protocol  Information 

1 

WhereNet  Experiment 

Resource  Allocation 

4 

WhereNet  Experiment 

Resource  Availability 

3 

WhereNet  Experiment 

Resource  Location 

3 

WhereNet  Experiment 

Resources  Needed 

1 

WhereNet  Experiment 

Schedule  Change 

5 

WhereNet  Experiment 

Swap 

8 

WhereNet  Experiment 

T/O  Time 

6 

WhereNet  Experiment 

Task  Assignment 

10 

WhereNet  Experiment 

Task  Clarification 

1 

WhereNet  Experiment 

Task  Time 

1 

WhereNet  Experiment 

Time  to  fix 

18 

WhereNet  Experiment 

Time  to  Site 

38 

WhereNet  Experiment 

Weather 

3 

3.2.2  Source  Frequency 

Where  the  participants  sought  information  was  documented  and  the  frequency  was 
tabulated.  Table  20  illustrates  this  data. 
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Table  20:  Source  Frequency  Table 


Condition 

Source 

Count  Of  Source 

Baseline  Experiment 

Baseline  Experiment 

AGE 

10 

Baseline  Experiment 

AMU  Dispatch 

10 

Baseline  Experiment 

Base  Map 

8 

Baseline  Experiment 

Crew  Chief 

1 

Baseline  Experiment 

Equipment  Resource  Sheets 

2 

Baseline  Experiment 

Expeditor 

5 

Baseline  Experiment 

Hydraulics  Shop 

1 

Baseline  Experiment 

Maintenance  Supervisor 

2 

Baseline  Experiment 

MOC 

6 

Baseline  Experiment 

Munitions 

5 

Baseline  Experiment 

Personnel  List 

1 

Baseline  Experiment 

Personnel  Sheet 

2 

Baseline  Experiment 

PI 

3 

Baseline  Experiment 

Weather  Card 

1 

Baseline  Experiment 

ProSuper 

1 

Baseline  Experiment 

Resource  Sheet 

5 

Baseline  Experiment 

Scenario/PI 

10 

Baseline  Experiment 

Self 

24 

Baseline  Experiment 

Specialist 

31 

Baseline  Experiment 

Status  Sheet/Schedule 

16 

Smart  Systems  Experiment 

0 

Smart  Systems  Experiment 

Add  Comparison:  Drop  Down 

2 

Smart  Systems  Experiment 

AGE 

1 

Smart  Systems  Experiment 

Base  Map  Paper 

1 

Smart  Systems  Experiment 

Comparison  Views 

49 

Smart  Systems  Experiment 

Context  View 

1 

Smart  Systems  Experiment 

Fix  Resources/  Problem  Schedule 

View 

1 

Smart  Systems  Experiment 

MOC 

2 

Smart  Systems  Experiment 

Overview  Geo/Schedule  View 

19 

Smart  Systems  Experiment 

Pop  Up  Box 

1 

Smart  Systems  Experiment 

problem  Geo  View 

5 

Smart  Systems  Experiment 

Problem  Schedule  View 

27 

Smart  Systems  Experiment 

Scenario 

2 

Smart  Systems  Experiment 

Schedule/Status  Sheet 

2 

Smart  Systems  Experiment 

Select  Plan 

11 

Smart  Systems  Experiment 

Self 

1 

Smart  Systems  Experiment 

Smart  Systems 

10 
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3.2.3  Destination  Frequency 

Frequencies  were  determined  for  Destination,  which  refers  to  where  the  participant 
sent  or  distributed  the  information.  This  information  can  be  found  in  Table  21 . 


Table  21 :  Destination  Frequency  Table 


Condition 


Baseline  Experiment 


Baseline  Experiment 


Baseline  Experiment 


Baseline  Experiment 


Baseline  Experiment 


Baseline  Experiment 


Baseline  Experiment 


Baseline  Experiment 


Baseline  Experiment 


Baseline  Experiment 


Baseline  Experiment 


Smart  Systems  Experiment 


Destination 
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Condition 

Destination 

Count  Of 
Destination 

Smart  Systems  Experiment 

AGE 

2 

Smart  Systems  Experiment 

AMU  Dispatch 

1 

Smart  Systems  Experiment 

MOC 

1 

WhereNet  Experiment 

WhereNet  Experiment 

AGE 

14 

WhereNet  Experiment 

AMU  Dispatch 

6 

WhereNet  Experiment 

Crew  Chiefs 

1 

WhereNet  Experiment 

Expeditor 

4 

WhereNet  Experiment 

Maintenance  Supervisor 

2 

WhereNet  Experiment 

MOC 

16 

WhereNet  Experiment 

Munitions 

4 

WhereNet  Experiment 

Pl/Scenario 

1 

WhereNet  Experiment 

ProSuper 

6 

WhereNet  Experiment 

Specialist 

0 
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Appendix  C 
SSLC2  Scenarios 
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1  Introduction 

SSLC2  Team  members  developed  a  series  of  scenarios  highlighting  common 
flightline  problems  requiring  various  personnel  and  equipment  resources.  The  scenarios 
were  developed  with  input  from  various  Subject  Matter  Experts  and  were  made  to  be  as 
realistic  as  possible.  These  scenarios,  detailed  in  this  Appendix,  became  the  basis  for  the 
Scientific  Study  simulation.  Team  members  created  four  distinct  scenarios  detailing  a 
fuel  leak,  gland  nut  crack,  engine  stall,  and  cockpit  glass  removal  each  tailored  for  both 
the  airlift  and  fighter  situations.  The  scenarios  and  associated  data  were  presented  to  the 
scientific  study  participants  through  a  script  and  set  of  procedures.  Related  data,  such  as 
personnel  and  resources,  were  presented  through  the  WhereNet  and  SSLC2  technology 
conditions  to  enhance  the  scenarios. 

2  Scenario  1 :  Left  Main  Strut  -  Airlifter  (Tail  #89-0449) 

During  the  walk-around  inspection,  the  crew  reports  hydraulic  fluid  on  the  left  main 
strut.  Thirty  minutes  earlier  the  Crew  Chief  had  inspected  the  struts  and  found  no  trace 
of  hydraulic  fluid  in  that  location.  The  Crew  Chief  calls  the  Expediter.  The  Expediter 
knows  that  in  the  worse-case  scenario,  they  will  need  to  change  the  strut.  In  order  to 
change  the  strut,  the  aircraft  will  need  to  be  defueled  and  jacked.  That  also  means  they 
will  need  a  power  unit,  a  hydraulic  mule,  a  set  of  jacks  (6  jacks),  a  jacking  team  (7 
people),  and  a  defuel  team  (3  people  can  be  some  of  the  same  people  as  the  jacking 
team),  a  defuel  truck,  and  a  jacking  manifold.  Also  required  will  be  a  three  person 
airfreight  crew  with  a  K-Loader  to  download  the  cargo,  plus  the  pneudraulics  technician 
to  do  the  actual  work  as  well  as  any  subsequently  discovered  bad  parts. 

During  troubleshooting  the  pneudraulics  technician  discovers  the  gland  nut  is  cracked 
and  must  be  changed  (jacking  and  gear  retraction  will  be  required).  He  informs  the 
Expediter  and  orders  the  part.  The  Expediter,  who  is  somewhat  relieved  that  the  strut 
itself  does  not  need  to  be  changed,  reports  the  problem  to  the  production  supervisor 
(ProSuper)  and  Maintenance  Operations  Center  (MOC)  requesting  airfreight  to  download 
the  cargo.  Cargo  download  is  a  rather  routine  process  and  everything  goes  as  expected. 
Without  waiting  for  air  freight  to  finish,  the  Expediter  orders  the  defuel  truck,  the  jacking 
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manifold,  jacks,  and  hydraulic  mule.  They  also  need  a  power  cart,  but  there  is  one 
already  at  the  aircraft  from  take-off  preparations.  While  air  freight  was  in  the  process  of 
removing  the  last  of  the  cargo,  the  Expediter  gets  word  the  power  cart  just  ran  out  of  fuel. 
Some  of  the  preparations  can  continue,  but  the  power  cart  will  be  needed  for  defuel. 
Airfreight  has  completed  their  job  in  about  20  minutes. 

Once  the  cargo  has  been  unloaded,  the  Expediter  drops  off  the  jack  team  and  instructs 
them  to  begin  preparing  for  the  jack  job  while  waiting  for  defuel.  The  jacks  can  be 
positioned  and  connected  to  the  jacking  manifold  and  the  pneudraulics  lines  from  the 
mule  can  be  connected  to  the  aircraft. 

The  fuel  truck  and  new  power  cart  finally  arrive  and  the  aircraft  defueled.  As  they 
begin  sorting  themselves  out  the  jack  crew  suddenly  realizes  that  none  of  them  is  jack 
supervisor  qualified.  The  Expediter  had  expected  the  TSgt  he  dropped  off  to  be  the 
supervisor,  but  the  TSgt  recently  crossed  over  from  another  airframe  and  has  not  yet  been 
certified  on  everything.  The  problem  now  is  that  everyone  else  on  the  flightline  seems  to 
be  busy  with  the  morning  launch  and  he  needs  to  find  another  individual  to  supervise  the 
jack  job  quickly! 

A  SSgt  Crew  Chief  washing  her  aircraft  at  the  wash  rack  is  jack  supervisor  qualified, 
so  the  TSgt  goes  quickly  to  relieve  her.  She  arrives  and  the  aircraft  is  jacked  in  short 
order.  While  the  jacking  commences,  the  Expediter  contacts  the  specialist  supervisor  to 
call  for  the  pneudraulics  specialist  to  come  out  and  get  to  work. 

Finally  the  pneudraulics  technician  takes  time  to  review  the  tech  data  and  identify 
future  road  blocks.  When  all  the  parts  are  assembled  and  the  aircraft  has  been  jacked  he 
and  the  Crew  Chief  begin.  Once  the  gland  nut  is  installed  it  is  pressure  checked  and  then 
a  gear  retraction  check  is  made.  Once  everything  is  determined  to  be  working  properly, 
the  aircraft  is  lowered  to  the  ground  and  returned  to  flying  condition. 

3  Scenario  1:  Left  Main  Strut  -  Fighter  (Tail  #85-1462) 

During  the  walk-around  inspection,  the  pilot  reports  hydraulic  fluid  on  the  left  main 
strut.  Thirty  minutes  earlier  the  Crew  Chief  had  inspected  the  struts  and  found  no  trace 
of  hydraulic  fluid  in  that  location.  The  Crew  Chief  calls  the  Expediter.  The  Expediter 
knows  that  in  the  worse-case  scenario,  they  will  need  to  change  the  strut.  This  means  that 
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they  will  need  to  jack  the  aircraft  and  because  of  the  heavy  fuel  load,  the  aircraft  will 
need  to  be  de-fueled  before  jacking.  This  job  will  require  a  power  unit,  a  hydraulic  mule, 
a  set  of  jacks  (3  jacks),  a  jacking  team  (4  people),  a  defuel  team  (3  people  can  be  the 
same  people  as  the  jacking  team),  the  defuel  truck,  plus  an  MC-7  cart  to  purge  the  drop 
tanks.  Also  required  will  be  the  weapons  load  crew  to  disarm  and  download  the  weapons 
as  well  as  any  subsequently  discovered  bad  parts. 

During  troubleshooting  the  Crew  Chief  discovers  that  the  gland  nut  is  cracked  and 
must  be  changed.  Even  though  the  strut  itself  is  good,  jacking  and  gear  retraction  will  be 
required  regardless.  The  Crew  Chief  informs  the  Expediter  and  orders  the  part.  The 
Expediter  has  the  weapons  load  crew  safe  the  gun  and  download  the  munitions.  This  is  a 
rather  routine  process  and  everything  goes  as  expected.  Without  waiting  for  the  load 
crew  to  finish,  the  Expediter  orders  the  defuel  truck,  the  MC-7  as  well  as  the  jacks  and 
hydraulic  mule.  They  also  need  a  power  cart  and  there  were  none  on  the  ready  line  a  few 
minutes  ago,  there  is  one  sitting  at  the  next  aircraft,  seemingly  unneeded  there.  While  the 
load  crew  was  in  the  process  of  removing  the  last  of  the  missiles,  the  Expediter  gets  the 
word  from  the  Crew  Chief  that  the  power  unit  they  pulled  over  from  the  adjacent  spot  is 
out  of  fuel.  The  load  crew  has  completed  their  job  in  about  20  minutes. 

Once  the  weapons  are  clear  and  the  new  power  unit  is  in  position,  the  Expediter  drops 
off  the  three  other  people  he  recruited  for  the  jack  team  and  instructs  them  to  begin 
preparing  for  the  jack  job  while  waiting  for  the  defuel.  The  jacks  can  be  positioned  and 
the  pneudraulics  lines  from  the  mule  can  be  connected  to  the  aircraft.  The  MC-7  is  also 
put  into  place  in  preparation  for  the  defuel  operation. 

The  empty  truck  arrives  and  the  aircraft  defueled.  The  jack  crew  suddenly  realizes 
that  none  of  them  is  jack  supervisor  qualified.  The  Expediter  had  expected  the  TSgt  he 
dropped  off  to  be  the  supervisor,  but  the  TSgt  recently  crossed  over  from  another 
airframe  and  has  not  yet  been  certified  on  everything.  The  problem  now  is  that  everyone 
on  the  flightline  seems  to  be  busy  with  the  morning  launch  and  he  needs  to  find  another 
individual  to  supervise  the  jack  job  quickly! 

The  Aircraft  Maintenance  Unit  (AMU)  dispatch  position  (a  rotating  duty)  is  currently 
being  manned  by  a  SRA  who  is  jack  supervisor  qualified  and  so  the  TSgt  quickly  goes 
into  the  AMU  and  relieves  her.  She  arrives  and  the  aircraft  is  jacked  in  short  order. 
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Finally  the  Crew  Chief  has  the  time  to  review  the  tech  data  to  identify  future  road 
blocks.  When  all  the  parts  are  assembled  and  the  aircraft  has  been  jacked  he  and  another 
Crew  Chief  begin  the  repair.  Once  the  gland  nut  is  installed  it  is  pressure  checked  and 
then  a  gear  retraction  check  is  made.  Once  everything  is  determined  to  be  working 
properly,  the  aircraft  is  lowered  to  the  ground  and  returned  to  flying  condition. 

4  Scenario  2:  Cockpit  Glass  -  Airlifter  (Tail  #  89-0452) 

During  the  Crew  Chiefs  preflight  cockpit  glass  cleaning,  she  notices  a  crack  along 
the  base  of  the  window.  The  crack  is  aligned  with  the  cockpit  glass  frame  so  as  to  make 
discovery  unlikely  unless  closely  inspected.  After  a  quick  check  of  the  technical  order 
(TO),  she  believes  that  the  crack  is  beyond  limits  and  the  glass  must  be  changed.  The 
Expediter  is  called  and  the  situation  is  assessed.  If  the  cockpit  glass  does  indeed  need  to 
be  changed,  they  will  have  to  tow  the  aircraft  into  a  hangar  as  changing  the  glass 
outdoors  is  not  an  option  today  with  the  wind  and  rain  currently  falling.  In  addition  to  the 
tow  tractor,  tow  bar  and  tow  crew  they  will  also  need  the  sheet  metal  technician  to 
evaluate  the  problem,  the  Aerospace  Repair  (AR)  technician  to  change  the  glass,  a  B-4 
Stand,  and  another  piece  of  glass.  Environmental  &  Electrics  (E&E)  technicians  will  be 
required  to  perform  a  continuity  check  on  the  window  heating  element  prior  to  sealing  the 
window.  A  heater  might  also  be  necessary  to  help  cure  the  sealant.  After  the  glass 
change  is  complete,  a  pressurization  cart,  and  the  E&E  technician  will  be  needed  to 
perform  the  pressure  check. 

Sheet  metal  technicians  determine  that  the  glass  must  be  changed.  They  inform  the 
Expediter  and  the  glass  is  ordered.  The  Expediter  checks  with  the  MOC  to  determine  a 
hangar  location  for  the  glass  change  since  inclement  weather  is  present.  The  Expediter 
orders  the  tow  operation  preparation  and  requests  the  AR  technicians  meet  the  aircraft  at 
the  hangar. 

The  tow  begins  and  the  aircraft  is  quickly  pushed  into  the  waiting  hangar.  AR  arrives 
with  the  new  glass  and  completes  the  removal  and  replacement  of  the  glass  and  calls  for 
E&E  to  come  perform  the  continuity  check.  E&E  arrive  and  begin  setting  up  for  the 
check  when  the  Specialist  Expediter  arrives  and  shouts  that  they  have  an  E&E  red  ball  on 
another  aircraft.  Enroute  the  Specialist  Expediter  calls  the  Crew  Chief  Expediter  to 
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suggest  he  look  for  one  of  the  other  E&E  troops  on  duty  to  ohm  out  the  window. 
Another  E&E  troop  arrives  and  the  continuity  check  is  completed. 

AR  applies  the  sealant  and  the  cure  begins.  There  is  a  heater  just  outside  the  hangar 
door  and  the  crew  chief  connects  the  hose  extenders  so  that  the  heater  can  be  used  to 
speed  the  cure.  After  just  a  few  minutes  the  heater  quits  and  although  there  is  plenty  of 
fuel,  it  just  won’t  start  again.  The  Crew  Chief  flags  down  the  specialist  truck  to  report 
the  situation  to  the  Expediter.  Another  heater  is  brought  out.  In  a  few  hours  the  sealant  is 
cured  and  the  aircraft  is  ready  for  pressure  testing.  The  Expediter  calls  for  the  E&E  tech 
and  the  pressurization  cart.  The  E&E  technician  makes  it  back  to  the  aircraft  and  the 
pressure  check  is  completed  with  no  discrepancies. 

5  Scenario  2:  Canopy  Glass  -  Fighter  (Tail  #  85-1459) 

During  the  Crew  Chiefs  preflight  canopy  cleaning,  she  notices  a  crack  along  the  base 
of  the  window.  The  crack  is  aligned  with  the  canopy  frame  so  as  to  make  discovery 
unlikely  unless  closely  inspected.  She  believes  that  the  crack  is  beyond  limits  and  the 
canopy  must  be  changed.  The  Expediter  is  called  and  the  situation  is  assessed.  If  the 
canopy  does  indeed  need  to  be  changed,  they  will  have  to  tow  the  aircraft  into  a  hangar  as 
changing  the  canopy  outdoors  is  not  reasonable  because  it  is  raining  and  the  wind  is 
blowing.  In  addition  to  the  tow  tractor,  tow  bar  and  tow  crew  they  will  also  need  the 
sheet  metal  technician  to  evaluate  the  problem  and  change  the  glass  (if  necessary),  egress 
technicians  to  disarm  the  ejection  seat  and  remove  the  canopy,  the  canopy  hoist,  the 
canopy  jig,  and  another  canopy  glass.  Also,  a  weapons  load  crew  will  be  needed  to 
disarm  the  aircraft  and  download  the  weapons.  After  the  canopy  change  is  complete,  a 
pressurization  cart  and  the  Environmental  &  Electrics  (E&E)  technician  will  be  needed  to 
perform  the  pressure  check. 

Sheet  metal  technicians  determine  the  canopy  must  be  changed.  They  inform  the 
Expediter  and  order  the  canopy.  The  Expediter  has  the  weapons  load  crew  safe  the  gun 
and  download  the  munitions.  This  is  a  rather  routine  process  and  everything  goes  as 
expected.  The  Expediter  checks  with  the  MOC  to  determine  a  hangar  location  for  the 
canopy  change.  Without  waiting  for  the  load  crew  to  finish,  the  Expediter  orders  the  tow 
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operation  preparation  and  requests  the  egress  technicians  and  the  canopy  hoist  meet  the 
aircraft  at  the  hangar. 

Once  the  weapons  are  clear,  the  tow  begins  and  the  aircraft  is  quickly  pushed  into  the 
waiting  hangar.  Egress  arrives  with  a  new  canopy,  already  built  up  from  the  shop.  They 
safe  the  seat  and  remove  the  canopy.  While  in  the  middle  of  lowering  the  old  canopy,  the 
Specialist  Expediter  drives  up  and  tells  the  egress  technician  that  he  is  needed  for  a  red 
ball  foreign  object  damage  (FOD)  in  the  cockpit  on  another  aircraft.  The  Specialist 
Expediter  calls  the  Expediter  to  inform  him  of  the  actions  and  suggests  two  possible 
alternative  egress  technicians  that  are  on  duty.  The  Expediter  locates  another  egress 
technician  and  gets  him  to  pickup  where  the  other  left  off.  The  reinstallation  and 
longeron  check  occur  without  any  problems.  No  cure  time  is  needed  because  they  were 
able  to  use  a  previously  built  up  canopy  so  the  aircraft  is  ready  for  pressure  testing.  The 
Expediter  had  previously  called  for  the  pressurization  cart  and  it  is  already  waiting  at  the 
aircraft.  He  now  calls  for  the  E&E  tech  to  pressure  check  the  aircraft.  E&E  get 
everything  set  up  and  begin  the  pressure  test,  however  the  internal  pressure  gauge  is  not 
moving.  They  flag  down  the  next  vehicle  with  a  radio  and  report  the  situation  to  the 
Expediter.  The  Expediter  arranges  for  a  new  pressure  cart  and  the  pressure  check  is 
completed  with  no  discrepancies. 

6  Scenario  3:  Engine  Prop  Damage -Airlift  (Tail  #89-0448) 

In  the  preflight  inspection  the  Crew  Chief  discovers  the  #3  engine  prop  is  not 
serviceable. 

When  the  Expediter  arrives  at  the  aircraft,  he  calls  the  engine  technician  to  begin 
looking  into  the  problem.  The  Expediter  knows  that  in  the  worse  case  the  prop  will  need 
to  be  changed.  This  will  require  a  new  prop,  an  empty  prop  dolly,  a  heavy  lift  crane,  a 
prop  change  crew,  a  B1  stand  and  finally  an  engine  run  which  will  require  a  power  unit, 
an  engine  run  crew  (could  be  the  same  people  as  the  prop  change  crew)  as  well  as  a  take¬ 
off  rated  thrust  (TRT)  rated  parking  spot  to  perform  the  run.  They  will  also  need  a  tow 
vehicle,  tow  bar  and  tow  team  to  get  the  aircraft  to  the  TRT  spot. 

The  Expediter  orders  the  B1  stand,  the  prop  dolly  and  crane  while  the  engine 
technicians  order  the  prop.  The  crane  is  positioned  and  the  prop  change  crew  begins 
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lowering  the  prop.  As  the  crane  is  lowering  the  prop,  a  hydraulic  line  on  the  crane  begins 
to  leak.  The  prop  is  lowered  safely,  but  now  the  crane  is  not  operable.  The  old  prop  is 
transferred  to  the  dolly  and  returned  to  the  shop.  The  crane  is  a  unique  asset  and  they 
only  brought  one  of  them.  Vehicle  maintenance  can  repair  the  crane,  but  it  will  take  them 
about  two  hours  to  do  so.  The  Expediter  knows  that  the  fighter  AMU  possesses  a  cobra 
crane  for  work  on  canopies  and  seats.  This  crane  is  no  where  near  as  strong  as  the  heavy 
lift  crane  they  were  using,  but  strong  enough  to  lift  the  prop.  The  Expediter  calls  the 
MOC  to  coordinate  the  use  of  the  crane. 

The  crane  arrives  and  the  prop  is  installed.  After  the  installation  is  complete  the 
aircraft  needs  to  be  towed  to  the  TRT  site  for  a  full  engine  run.  The  tow  tractor  and  tow 
team  are  assembled  and  the  aircraft  is  towed.  The  aircraft  is  positioned  and  the  run  crew 
is  assembled.  As  they  obtain  clearance  for  the  run,  the  MOC  calls  the  Expediter  telling 
him  that  one  of  his  engine  run  people  has  an  emergency  at  home.  They  have  just 
received  a  Red  Cross  message  and  he  needs  to  be  on  the  next  flight  headed  back  to  the 
states.  The  engine  technician  is  returned  to  his  hotel  room  and  the  Expediter  begins 
looking  for  another  engine  run  qualified  technician.  Eventually,  a  Crew  Chief  with  the 
required  qualification  is  located  and  the  run  begins.  The  TRT  engine  run  is  completed 
and  the  aircraft  is  returned  to  operational  status. 

7  Scenario  3:  Engine  Compressor  Stall  -  Fighter  (Tail  #85- 
1464) 

During  an  80%  engine  run  for  scheduled  maintenance,  the  engine  exhibited  a  series  of 
loud  compressor  stalls.  This  aircraft  is  on  toady’s  flying  schedule.  The  Expediter  knows 
that  in  the  worse  case  the  engine  will  need  to  be  changed.  This  will  require  a  new  engine, 
a  3000  trailer,  an  engine  change  crew,  a  structural  repair  technician  to  perform  the  engine 
bay  inspection  prior  to  installing  the  new  engine,  and  finally  an  engine  run  which  will 
require  a  power  unit,  an  engine  run  crew  (could  be  the  same  people  as  the  engine  change 
crew)  as  well  as  the  trim  pad  to  perform  the  run.  Also  required  will  be  the  weapons  load 
crew  to  safe  the  aircraft  and  possibly  download  the  weapons. 

Upon  troubleshooting,  the  engine  technicians  discover  a  fuel  flow  regulator  has 
failed.  This  change  will  require  a  rather  simple  fix,  but  will  necessitate  an  afterburner  run 
after  the  regulator  is  changed.  The  Expediter  has  the  weapons  load  crew  safe  the  gun  and 
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download  the  munitions.  This  is  a  rather  routine  process  and  everything  goes  as 
expected.  The  engine  technicians  order  the  regulator.  Once  the  weapons  are  clear,  the 
Crew  Chief  begins  opening  the  panels  required  for  the  regulator  change.  While  doing  so, 
he  strips  a  screw  and  needs  structural  repair  to  assist  in  removing  the  screw. 

The  structural  repair  technician  needs  a  low  pressure  cart  (Low  pack)  in  order  to 
complete  the  job.  The  Expediter  checks  the  ready  line  and  collects  the  low  pack  and 
delivers  it  to  the  aircraft.  The  structural  repair  technician  was  just  getting  started  when  the 
first  sergeant  came  out  and  pulled  him  off  the  job.  The  squadron  had  received  a  Red 
Cross  message  explaining  that  his  little  girl  was  quite  sick  and  his  presence  was  requested 
back  home.  A  structural  repair  technician  is  pulled  from  the  other  aircraft  and  makes  the 
required  repairs  and  the  new  regulator  is  installed.  After  the  installation  is  complete  the 
aircraft  needs  to  be  towed  to  the  trim  pad  for  a  full  after  burner  (AB)  power  engine  run. 
The  tow  tractor  and  tow  team  are  assembled  but  the  tow  bar  is  missing.  The  unit 
deployed  more  tow  tractors  than  tow  bars  as  the  tractors  are  used  for  towing  more  than 
just  aircraft.  Eventually  the  Expediter  finds  a  tow  bar  near  the  hangar  and  the  tow  is 
allowed  to  proceed.  The  aircraft  is  appropriately  secured  and  the  run  crew  is  assembled. 
The  AB  engine  run  is  completed  and  the  aircraft  is  returned  to  operational  status. 

8  Scenario  4:  Fuel  Leak  -  Airlift  (Tail  #89-0447) 

The  Crew  Chief  discovers  fuel  leaking  from  the  wing  root  area.  The  leak  is  not  bad 
and  has  not  created  a  fuel  spill.  Fuels  technicians  use  a  quick  access  panel  to  determine 
the  leak  is  beyond  limits  and  the  transfer  valve  must  be  changed.  Prior  to  changing  the 
valve,  the  aircraft  fuel  level  must  be  brought  below  the  level  of  the  valve  and  the  aircraft 
must  be  towed  into  the  fuel  cell.  Since  the  valve  is  external  to  the  tank,  tank  entry  will 
not  be  required.  After  the  valve  is  changed,  the  aircraft  must  again  be  towed,  fuel 
brought  back  up  to  the  appropriate  level  for  the  check  and  then  a  leak  and  transfer  check 
must  be  performed.  Needed  for  this  job  will  be  the  fuels  technician,  the  tow  bar,  tow 
tractor,  tow  crew,  a  low  pack  to  aid  in  depaneling  the  aircraft,  and  when  the  aircraft 
returns  to  the  flightline,  a  power  unit  and  a  light  cart  will  be  needed  for  the  leak  and 
transfer  checks  which  will  likely  occur  during  hours  of  darkness. 


150 


The  fuels  technician  provides  configuration  requirements  for  the  aircraft  to  the 
Expediter  and  returns  to  the  fuel  cell  (shop)  to  order  the  part.  The  configuration 
requirements  include  (among  other  things)  depaneling  the  wing  root  as  well  as 
disconnecting  the  battery.  Two  Crew  Chiefs  are  working  the  configuration  checklist,  one 
removing  the  requisite  panels  and  the  other  doing  the  other  prep  work.  While 
disconnecting  the  batteries,  the  Crew  Chief  somehow  shorts  out  the  battery  which  results 
in  minor  bums  on  the  Crew  Chiefs  hands.  The  Specialist  Expediter  was  driving  by  at 
the  time  and  takes  the  injured  Crew  Chief  to  the  hospital.  The  other  Crew  Chief  notifies 
the  Expediter  who  reports  the  incident  to  the  MOC  and  begins  to  look  for  a  replacement 
Crew  Chief  to  complete  the  job.  The  battery  condition  will  be  evaluated  when  the 
aircraft  returns  to  the  flightline.  The  new  Crew  Chief  arrives  and  the  configuration 
checklist  is  completed.  The  Expediter  organizes  the  tow  crew  and  ensures  the  tow  tug 
and  tow  bar  are  available.  The  aircraft  is  towed  to  the  fuel  cell  and  the  work  is  done 
quickly.  Once  the  job  is  complete,  the  fuel  cell  notifies  the  MOC  to  let  the  Expediter 
know  to  return  the  aircraft  to  the  flightline.  The  Expediter  ensures  the  tow  equipment  and 
crew  head  back  to  the  fuel  cell  and  the  aircraft  is  returned  to  the  line  in  short  order.  The 
Crew  Chief  inspects  the  battery  and  it  appears  to  be  serviceable  and  power  on  confirms 
that  it  was  undamaged.  The  leak  and  transfer  checks  begin.  The  leak  check  seems  to  be 
good,  but  the  Hobart  power  unit,  used  to  provide  the  power  needed  to  perform  the 
transfer  check,  keeps  kicking  the  circuit  off-line.  Hobarts  have  a  tendency  to  not  work 
properly  with  certain  airplanes  and  this  particular  one  and  aircraft  447  do  not  seem  to  get 
along.  A  new  Hobart  is  needed.  The  new  Hobart  arrives  and  seems  to  work  properly 
with  the  aircraft  systems.  The  leak  and  transfer  checks  work  properly  and  the  aircraft 
returns  to  fully  mission  capable  status. 

9  Scenario  4:  Fuel  Leak  -  Fighter  (Tail  #85-1463) 

While  performing  preflight  inspections,  the  Crew  Chief  discovers  fuel  leaking  from 
the  wing  root  area.  The  leak  is  not  bad  and  has  not  created  a  fuel  spill.  Fuels  technicians 
use  a  quick  access  panel  to  determine  that  the  leak  is  beyond  limits  and  the  transfer  valve 
must  be  changed.  Prior  to  changing  the  valve,  the  aircraft  fuel  level  must  be  brought 
below  the  level  of  the  valve  and  the  aircraft  must  be  towed  into  the  fuel  cell.  After  the 
valve  is  changed,  the  aircraft  must  again  be  towed,  fuel  must  be  brought  back  up  to  the 
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appropriate  level  for  the  check  and  then  a  leak  and  transfer  check  must  be  performed. 
Needed  for  this  job  will  be  the  fuels  technician,  the  tow  bar,  tow  tractor,  tow  crew,  a  low 
pack  to  aid  in  depaneling  the  aircraft,  and  when  the  aircraft  returns  to  the  flightline,  a 
power  unit,  a  C-10  air  conditioner  and  a  light  cart  will  be  needed  for  the  leak  and  transfer 
checks  which  will  likely  occur  during  hours  of  darkness. 

The  fuels  technician  provides  a  configuration  checklist  for  the  aircraft  to  the 
Expediter  and  returns  to  the  fuel  cell  (shop)  to  order  the  part.  The  configuration 
requirements  include  depaneling  the  wing  root  as  well  as  disconnecting  the  battery  and 
other  preparatory  work.  Two  Crew  Chiefs  are  working  the  configuration  checklist,  one 
removing  the  requisite  panels  and  the  other  doing  the  other  prep  work.  While 
disconnecting  the  battery,  the  Crew  Chief  somehow  shorts  out  the  battery  which  results  in 
minor  burns  on  his  hands.  The  Specialist  Expediter  was  driving  by  at  the  time  and  takes 
the  injured  Crew  Chief  to  the  hospital.  The  other  Crew  Chief  notifies  the  Expediter  who 
reports  the  incident  to  the  MOC  and  begins  to  look  for  a  replacement  Crew  Chief  to 
complete  the  job.  The  battery  condition  will  be  evaluated  when  the  aircraft  returns  to  the 
flightline.  The  new  crew  Chief  arrives  and  the  configuration  checklist  is  completed.  The 
Expediter  organizes  the  tow  crew  and  ensures  the  tow  tug  and  tow  bar  are  available.  The 
aircraft  is  towed  to  the  fuel  cell  and  the  work  is  done  quickly.  Once  the  job  is  complete, 
the  fuel  cell  notifies  the  MOC  to  let  the  Expediter  know  to  return  the  aircraft  to  the 
flightline.  The  Expediter  ensures  the  tow  equipment  and  crew  head  back  to  the  fuel  cell 
and  the  aircraft  is  returned  to  the  line  in  short  order.  The  Crew  Chief  inspects  the  battery 
and  it  appears  to  be  serviceable  and  power  on  confirms  that  it  was  undamaged.  The  leak 
and  transfer  checks  begin.  The  leak  check  seems  to  be  good,  but  the  -60  power  unit  used 
to  provide  the  power  needed  to  perform  the  transfer  check,  is  not  putting  out  enough 
bleed  air  to  properly  drive  the  C-10.  The  C-10  is  to  provide  auxiliary  cooling  for  the 
avionics  and  electrical  systems  when  the  aircraft  air  conditioner  is  not  operating.  A  new  - 
60  is  needed.  The  new  -60  arrives  and  seems  to  work  properly  with  the  aircraft  systems. 
The  leak  and  transfer  checks  work  properly  and  the  aircraft  returns  to  fully  mission 
capable  status. 
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